Let’s pick(et) our battles wisely

VROOMFONDEL: We demand that machine not be allowed to think about this problem!

DEEP THOUGHT: If I might make an observation…

MAJIKTHISE: We’ll go on strike!

VROOMFONDEL: That’s right. You’ll have a national philosophers’ strike on your hands.

DEEP THOUGHT: Who will that inconvenience?

MAJIKTHISE: Never you mind who it’ll inconvenience you box of black legging binary bits! It’ll hurt, buster! It’ll hurt!

     From Fit The Fourth of The Hitch-Hiker’s Guide To The Galaxy, Douglas Adams.  Broadcast on BBC Radio 4, March 29 1978.


I suspect that this is going to be a contentious post.

Having spent my time on the picket lines over the last eight (non-)working days…

…and last year,

… I am acutely aware of, and deeply sympathetic to, the issues underpinning the strike. The speeches at yesterday’s closing rally — including that from the ever-impressive Lilian Greenwood, Labour MP for Nottingham South (and someone for whom I will again be voting in a week’s time) — brought home the exceptionally precarious and deeply unfair working conditions that so many university employees endure under zero hours contracts. Even Spiked! magazine — whose coverage of universities usually fixates on hysterical fantasies about the infestation of evil, leftist, free-speech-suppressing, no-platforming Cultural Marxists indoctrinating our children — saw fit to publish a rousing article supporting the strikes.

There has similarly been a series of compelling and affecting pieces over the last few weeks that drive home the damage that the ever-accelerating corporatisation and marketisation of our universities is doing to education. One of the more comprehensive analyses I’ve seen is The Seven Deadly Sins of Marketisation in British Higher Education by Lee Jones, Reader in International Politics at Queen Mary University of London. Thoroughly recommended.

But what have these eight days on strike actually achieved?

Yes, I know that we’ve demonstrated a great deal of solidarity and that the time on the picket lines has been morale-boosting (and at least it wasn’t as sodding cold as last year). But still, pragmatically, what did we achieve?

Here at Nottingham, at least, the response from the “powers that be” has been a deafening silence. (And Nottingham’s hardly alone in this.) For many departments, including my own, it’s been business as usual; the car park has been full, lectures and lab sessions went ahead with nary a disturbance, and coursework was dutifully marked and returned to students. This is not to downplay in any way, I hasten to add, the heartening efforts of my UCU colleagues and our incredibly supportive students, including, in particular, those who occupied UoN’s iconic Trent Building…

And I’ve also got to highlight the incredible energy, charisma, and tenacity of Matt Green, the President of Nottingham’s UCU Committee, who has been as outstanding as ever.

But the upshot of our eight day strike is that …drum roll… the UCU is going to call for yet more strikes in January. The argument is that we’ve got to keep the pressure up. But who, exactly, are we pressuring? Or, as Deep Thought puts it in that salient quote that opens this post, who, exactly, are we inconveniencing? We’ve hardly brought senior university management to their knees, have we?

For those who, like me, were on the picket lines — and, indeed, for those who weren’t — ask yourself this: which of the options below hurts the university more? Which is more likely to cause some sleepless nights for the senior executive?

A. An empty seminar room or lecture theatre,

B. A five- or ten-strong picket line chanting at a university entrance,

or,

C. A low score in the National Student Survey/ low league table ranking/ damaging media coverage for their university?

Not only did we have PVCs and other senior staff crossing picket lines with wild abandon, but quite a few union members — and, indeed, some erstwhile union reps — didn’t strike, let alone picket. University management will be well aware of this lack of engagement with the strike either now or when the figures for non-pay in January are returned. They save on the salary bill and they can rest easy that the impact on students’ progression is minimal, at best, and negligible, at worst.

Because what most matters to universities is their brand. If we want to have greater influence and bargaining power I would argue that we have to be a little more canny in our tactics and exploit exactly the corporatisation and marketisation culture we criticise and that underpins the behaviour of the 21st century university. (I’ve written before about the frustrating tendency of the left to not always be entirely cognisant of the value of “optics” and PR.)

Sceptical? Here are a few examples of brand management that might help to make my case…

Along with a number of APM colleagues, I spent six months chasing up a (very modest) honorarium payment for an invited speaker. Six months. The speaker eventually reached the point where, exasperated, she tweeted about the University’s lack of payment to her tens of thousands of followers (tagging in @UniOfNottingham). Within minutes she had a response from UoN, and within days the money was in her account.

Down the road, at Nottingham-Trent University (Guardian University Of The Year 2019), Liz Morrish was subject to disciplinary proceedings when a post hit 10,000 views on Liz’s own blog and trended at the Times Higher website, as described in the article linked in the tweet below.

And Warwick hardly covered itself in glory in this appalling case because they placed their brand management well ahead of students’ safety. That’s how engrained the importance of protecting the university brand can be.

“The top six universities are like the most beautiful cities in the world, reputable even if they have failing ­sewers, arrogant mayors and dodgy no-go areas…A folklore builds up around them, as do money and fans.”

(From Beyond the super-brands, universities are strengthening their positions, Times Higher Education)

So let’s stop trying to repeatedly use the same seventies strategies to attack a 21st century problem. Let’s think a little bit more about what really matters to university managers.

It’s not the students*.

It’s not the staff.

It’s the brand.  


 

* …although it’s certainly the student numbers.

Peterson, Scepticism, and the Art of Persuasion

I’m writing this from the not-so-sunny climes of Maidstone West station, waiting for the 09:03 to take me to London St Pancras (via the wonderfully-named Strood), from which I’ll get the train to Nottingham. I’m travelling back from Maidstone after a Skeptics In The Pub talk last night in the Market House pub there. It’s been over three years since my last visit to Maidstone SITP, which was again at the invitation of Rob Millar, the local Skeptics… coordinator. Thank you, Rob, for the invitation, the hospitality, sorting out accommodation, locating a guitar amplifier at the eleventh hour [1], and your careful chairing of what became a rather “robust” Q&A session at the end of the night.

I promised Rob and the other SITP regulars that I’d upload the slides I used, so here they are:

 

The majority of the talk was focused on “Uncertainty To 11…” themes, and I was delighted that Maidstone Skeptics asked very many perceptive, smart, and challenging questions about the nature of the quantum world (and much more). I just hope that my lengthy discourse on spatial frequencies through the medium of Stryper‘s sartorially-challenged stage attire was not the cause of too much indigestion last night. (Extra brownie points to the SITP regular who correctly identified both Stryper and Bad News [2] from the photos in my presentation. Clearly a man who, like myself, knows a little too much about eighties metal…)

Given that this was a Skeptics crowd, I felt obliged to include a couple of diversions from the quantum-meets-metal theme on the nature of skepticism, the devaluation of expertise, and the hysteria and hypocrisy of certain reactionary factions/fanatics in relation to university education. Those gender studies and lefty sociology courses mean that the nation is doomed, don’t you know…?

Hat tip to Tony Padilla for making me aware of Lance’s over-excited tweet above. Mr. Forman doubles, triples, and quadruples down on his pearl-clutching in a series of increasingly hyperbolic responses, including this:

Lance’s call to root out opinions he doesn’t agree with in order to, ahem, protect free speech — as one uber-reactionary pundit would put it, you can’t make this up — isn’t, of course, an entirely original demand. Jordan Peterson, along with other members of the self-styled Intellectual Dark Web — stop sniggering at the back there — has been howling for academics’ heads on a plate for quite a number of years because they simply will not toe the line, do as they’re told, and goddamn teach his preferred doctrines.

Mention of Peterson’s self-help psychobabble last night (see Slide #17 above) led to quite a heated discussion in the Q&A session following the talk. Three years ago I spent quite a bit of time lampooning Deepak Chopra’s “quantum woo”to a receptive Skeptics audience in Maidstone. What I find so difficult to get my head around is that very many of those who would identify as “rational skeptics” (or similar), and who rightly dismiss Chopra’s fairy tales out of hand, also represent a significant proportion of Peterson’s core fanbase. And yet, as I discussed at length in a talk for Nottingham’s Agnostic, Secularist, and Humanist society last year, Peterson’s “12 Rules For Life” and “Maps of Meaning” push the bullshit meter just as far above 11 as anything Chopra has written. Peterson’s style-over-substance, read-into-it-whatever-you-like, self-help gobbledegook also, hilariously, has very much in common with the wilfully impenetrable junk that is produced by the worst of the postmodernists he so despises.

Although there have been very many forensic dissections and demolitions of Peterson’s purple prose — with both this and this worthy of special mention — it was Private Eye that really got the measure of the man in a pitch-perfect parody of the vacuity of his writing:

Peterson_small

One important difference between Peterson and Chopra, however, is that the latter, while sharing Peterson’s charisma, oratory flair, and style-over-substance shtick, is not a poster boy for the worst type of reactionary right wing fervour, misogynistic movements (incels, in particular), and transphobic hate groups. Nor does Chopra, to the best of my knowledge, share, support, and help disseminate and normalise the views of Viktor Orbán [3], the infamously authoritarian Prime Minster of Hungary who is waging war on liberal values and shutting down university courses that don’t align with his personal ideological preferences. A recent article in the New York  magazine nailed it (and puts all of Peterson et al.‘s hand-wringing about no-platforming, “cancel culture”, and the like in context):

… if you are going to popularize the idea that leftist academics and human-rights organizations are poisoning the minds of children, and fomenting a subversive ideology antithetical to the health of your nation, then you simply cannot meet with an authoritarian prime minister who has used nearly identical arguments to justify state crackdowns on independent universities and NGOs — then issue no public explanation of why you took this meeting or objection to reports characterizing your conversation as convivial — and call yourself a principled defender of liberal values.

And to hammer it home:

Meanwhile, it isn’t hard to see how Peterson and Orbán might see eye to eye. The latter has effectively banned “gender studies” from his nation’s universities, while the former has called on his nation to do the same. What’s more, in a diatribe that Orbán’s speechwriters may wish to crib from, Peterson went so far as to suggest that left-wing instructors at a Canadian teachers college should be prosecuted for crimes against the state.

Lance Forman’s tweet above looks positively moderate in this context.

The central problem, however, is that the cult of personality surrounding Peterson (and, indeed, Chopra) is such that counter-arguments, data, and evidence are not going to sway those who feel that the great man has personally “spoken” to them via “12 Rules For Life” (or, in Chopra’s case, “The Seven Spiritual Laws for Success“) and changed them for the better. I asked the following question last night of the SITP regular who was a fan of Peterson:

“You say that Jordan Peterson’s “12 Rules For Life” spoke to you and made a difference in your life. How? Can you give me a specific example of something he wrote that had such an impact on you?”

“……..”

This is now the third time this has happened during a Q&A session. I don’t like putting people on the spot but I find it fascinating that when asked to highlight just one instance of Peterson’s writings that made a difference, each time I get a blank response to that question. This is entirely in line with Peterson’s writing style. He resonates with so many because, as Nathan Robinson explains so well in his classic take-down, Peterson’s writing is so nebulous and unclear that the reader takes their own meaning from the text. To be fair to Peterson, that type of writing takes a particular kind of skill. I only wish I’d realised this long before now [4] but it’s eerily reminiscent of the boilerplate that drives the horoscope market, as described in a classic Physics World article by Iggy McGovern [5]: Aspects of Low Resolution Horoscopy.

This leaves us with a conundrum. If even a hardened sceptic — an atheist/agnostic who rejects the likes of Deepak Chopra’s woo, for example — is taken in by Peterson’s guff, how can they be persuaded to be just a little more, um, sceptical (or, indeed, skeptical)? I am not at all suggesting that my approach last night — a rather full-on lampooning of Peterson — is any way to reach across the aisle, cathartic and fun though it was. Moreover, as an academic whose political leanings are left of centre, I will often be seen as one of the enemy. It is therefore going to be difficult, if you’ll excuse the understatement, to convince the Peterson faithful– whose numbers, I am willing to bet, include Lance up there — that I am not seeking to indoctrinate their children/ cause the collapse of Western civilisation/ establish a Cultural Marxist collective where it will be an ABSOLUTE PRIORITY to outlaw right wing views (delete/expand to taste).

So how do we connect? If we want to get beyond preaching to the converted, we obviously have to first find common ground with those who don’t share our political/ideological mindset. That’s tricky. But for the less evangelical of Peterson’s flock, science might be a way in. Or music. Or, indeed, both. There’s this book I could recommend…


[1] Thanks also to Ben for providing said amplifier.

[2] If you’re a Spinal Tap fan and you haven’t seen either of The Comic Strip Bad News specials, beg, borrow, or download the episodes asap. I guarantee that you won’t be disappointed. Sample quote, from Bad News’ lead guitarist and singer, Vim Fuego (aka Alan Metcalfe): “ I could play “Stairway To Heaven” when I was 12. Jimmy Page didn’t actually write it until he was 22. I think that says quite a lot.”

And here’s the Bad News boys in action…

[3] Thanks to Rob for bringing my attention to Peterson’s meeting with Orbán.

[4] Thanks, Lori, for drawing this parallel with horoscopy.

[5] Iggy was the external examiner for my PhD. He’s a poet as well as a physicist so knows a thing or two about writing style. This video with Iggy was a lot of fun to make back in (gulp) 2011…

What Is “University”?

On Thursday I’m going to be a member of a panel discussion, as part of a broader event, on the convoluted and compelling question of the fundamental role of universities in society:

What Is University?

Thu, 7 November 2019. 16:00 – 19:00 GMT. Teaching & Learning Building, UP.

This event is for academic and administrative staff, students and alumni of the University of Nottingham and asks:

  • What do you think is the purpose of a university?
  • What do you think it should be and might be in the future?

This is your opportunity to hear from senior figures with varying administrative and academic roles in the University of Nottingham, ask your questions, and share your views.

4 pm Afternoon tea and display of current students’ videos of What is ‘University’? – (Atrium, Teaching and Learning Building)

5-7pm Panel conversation, chaired by Prof Jeremy Gregory, FPVC for Arts:

  • Prof Shearer West, Vice-Chancellor
  • Prof Pam Hagan, Senior Tutor and Director of Student Well-being, School of Medicine
  • Dr Paul Greatrix, University Registrar
  • Prof Philip Moriarty, School of Physics and Astronomy
  • Mr Andrew Winter, Campus Life Director
  • Ms Stacy Johnson, School of Health Sciences and Deputy Hall Warden
  • Prof Peter Stockwell, School of English

Long gone are the halcyon days of dreaming spires and ivory towers — if they ever existed outside the less-than-entirely-fevered imaginations of a certain class of academic — and universities are increasingly being forced to question their place in the world…

WeDontNeedNoEducation.png

I’m looking forward to the event on Thursday — it promises to be a timely exploration of just why it is that academics do what they do. For homework (of my own), I’m going to ask some of those best placed to tell me about the role and function of a university education: our students. Today’s Politics, Perception, and Philosophy of Physics (PPP) session is going to focus on the “What Is “University”?” question. I’ll report back to the panel during Thursday’s event on the students’ feedback (and will write a follow-up post in due course.) For now, here are my slides for today’s PPP session. (As ever, however, the majority of the time will be given over to student discussion and debate.)


Update Nov 5 2019

The students provided a great deal of fascinating feedback and insightful contributions on this topic… (More on what they had to say in a future post.)

 

 

How To Write Your PhD Thesis Without Going Insane

Next Wednesday (Nov 6) the School of Physics & Astronomy will host a lunchtime seminar — at 1:00 pm in C12 in the main Physics Building — given by James Hayton, with the wonderfully descriptive title of “How To Write Your PhD Thesis Without Going Insane”. If my rapidly depleting and deteriorating memory doesn’t fail me, it’s been four years too long since Dr. Hayton last visited the School. I had the honour of supervising James’ (far-from-unchallenging*) PhD work and I always look forward immensely to his talks: engaging, entertaining, and essential listening for PhD students researchers and their supervisors alike**.

Here’s a brief description of the talk:

Writing is an essential skill for any PhD student (or professional academic). But writing can also be a significant source of stress. In fact, stress is so common that many people assume that it’s supposed to be stressful and you just have to suffer your way through.

But one of the reasons why writing is seen as so stressful is that very few people are trained to do it well. With the right approach, you can transform your writing from a barrier to work through into a powerful tool to help you communicate your research

In this talk, you’ll learn 3 key aspects of writing to help you communicate clearly and confidently, write a better thesis, faster, and maybe even enjoy the process.

…and a bio from James himself:

I completed my PhD in Physics here at Nottingham way back in 2007. Unlike many of my colleagues, I actually enjoyed the writing process, not only finishing writing in just 3 months, but  passing my viva with zero corrections.

I went on to two postdoc contracts in France and Spain before starting to coach PhD students in 2010. Since then, I’ve worked with hundreds of individual students and trained thousands more through webinars and online courses. I also published “PhD: an uncommon guide to research, writing & PhD life” in 2015.

Here’s James in action at Edinburgh five years ago. On Wednesday, I’m told that we’ll be getting a new, improved, revised, and revigorated version…***


*James was always a pleasure to work with, even during the most frustrating moments. It was the physics and instrument design/construction that were the challenging bits…

** Although James and I have ever-so-slightly diverging views on the value of a mock viva

*** …of the talk. (And possibly James.)

Guilty Confessions of a REFeree

#4 of an occasional series

At the start of this week I spent a day in a room in a university somewhat north of Nottingham with a stack of research papers and a pile of grading sheets. Along with a fellow physicist from a different university (located even further north of Nottingham), I had been asked to act as an external reviewer for the department’s mock REF assessment.

I found it a deeply uncomfortable experience. My discomfort had nothing to do, of course, with our wonderfully genial hosts — thank you all for the hospitality, the conversation, the professionalism, and, of course, lunch. But I’ve vented my spleen previously on the lack of consistency in mock REF ratings (it’s been the most-viewed post at Symptoms… since I resurrected the blog in June last year) and I agreed to participate in the mock assessment so I could see for myself how the process works in practice.

Overall, I’d say that the degree of agreement on “star ratings” before moderation of my co-marker’s grading and mine was at the 70% level, give or take. This is in line with the consistency we observed at Nottingham for independent reviewers in Physics and is therefore, at least, somewhat encouraging. (Other units of assessment for Nottingham’s mock REF review had only 50% agreement.)  But what set my teeth on edge for a not-insignificant number of papers — including quite a few of those on which my gradings agreed with those of my co-marker — was that I simply did not feel at all  qualified to comment.

Even though I’m a condensed matter physicist and we were asked to assess condensed matter physics papers, I simply don’t have the necessary level of hubris to pretend that I can expertly assess any paper in any CMP sub-field. The question that went through my head repeatedly was “If I got this paper from Physical Review Letters (or Phys. Rev. B, or Nature, or Nature Comms, or Advanced Materials, or J. Phys. Chem. C…etc…) would I accept the reviewing invitation or would I decline, telling them it was out of my field of expertise?”  And for the majority of papers the answer to that question was a resounding “I’d decline the invitation.”

So if a paper I was asked to review wasn’t in my (sub-)field of expertise, how did I gauge its reception in the relevant scientific community?

I can’t quite believe I’m admitting this, given my severe misgivings about citation metrics, but, yes, I held my nose and turned to Web of Science. And citation metrics also played a role in the decisions my co-marker made, and in our moderation. This, despite the fact that we had no way of normalising those metrics to the prevailing citation culture of each sub-field, nor of ranking the quality as distinct from the impact of each paper. (One of my absolutely favourite papers of all time – a truly elegant and pioneering piece of work – has picked up a surprisingly low number of citations, as compared to much more pedestrian work in the field.)

Only when I had to face a stack of papers and grade them for myself did I realise just how exceptionally difficult it is to pass numerical judgment on a piece of work in an area that lies outside my rather small sphere of research. I was, of course, asked to comment on publications in condensed matter physics, ostensibly my area of expertise. But that’s a huge field. Not only is no-one a world-leading expert in all areas of condensed matter physics, it’s almost impossible to keep up with developments in our own narrow sub-fields of interest let alone be au fait with the state of the art in all other sub-fields.

So we therefore turn to citations to try to gauge the extent to which a paper has made ripples — or perhaps even sent shockwaves – through a sub-field in which we have no expertise. My co-marker and I are hardly alone in adopting this citation-counting strategy. But that’s of course no excuse — we were relying on exactly the type of pseudoquantitative heuristic that I have criticised in the past and I felt rather “grubby” at the end of the (rather tiring) day. David Colquhoun made the following point time and again in the run up to the last REF  (and well before):

All this shows what is obvious to everyone but bone-headed bean counters. The only way to assess the merit of a paper is to ask a selection of experts in the field.

Nothing else works.

Nothing.

Bibliometrics are a measure of visibility and “clout” in a particular (yet often nebulously defined) research community; they’re not a quantification of scientific quality. Therefore, very many scientists, and this most definitely includes me, have deep misgivings about using citations to judge a paper’s — let alone a scientist’s — worth.

Although I agree with that quote from David above, the problem is that we need to somehow choose the correct “boundary conditions” for each expert; I can have a reasonable level of expertise in one sub-area of a field — say, scanning probe microscopy or self-assembly or semiconductor surface physics — and a distinct lack of working knowledge, let alone expertise, in another sub-area of that self-same field. I could list literally hundreds of topics where I would, in fact, be winging it.

For many years, and because of my deep aversion to simplistic citation-counting and bibliometrics, I’ve been guilty of the type of not-particularly-joined-up thinking that Dorothy Bishop rightly chastises in this tweet…

We can’t trust the bibliometrics in isolation (for all the reasons (and others) that David Colquhoun lays out here), so when it comes to the REF the argument is that we have to supplement the metrics with “quality control” via another round of ostensibly expert peer review. But the problem is that it’s often not expert peer review; I was certainly not an expert in the subject areas of very many of the papers I was asked to judge. And I’ll hold that no-one can be a world-leading expert in every sub-field of a given area of physics (or any other discipline).

So what are the alternatives?

David has suggested that we should, in essence, retire what’s known as the “dual support” system for research funding (see the video embedded below): “…abolish the REF, and give the money to research councils, with precautions to prevent people being fired because their research wasn’t expensive enough.” I have quite some sympathy with that view because the common argument that the so-called QR funding awarded via the REF is used to support “unpopular” areas of research that wouldn’t necessarily be supported by the research councils is not at all compelling (to put it mildly). Universities demonstrably align their funding priorities and programmes very closely with research council strategic areas; they don’t hand out QR money for research that doesn’t fall within their latest Universal Targetified Globalised Research Themes.

Prof. Bishop has a different suggestion for revamping how QR funding is divvied up, which initially (and naively, for the reasons outlined above) I found a little unsettling. My first-hand experience earlier this week with the publication grading methodology used by the REF — albeit in a mock assessment — has made me significantly more comfortable with Dorothy’s strategy:

.”..dispense with the review of quality, and you can obtain similar outcomes by allocating funding at institutional level in relation to research volume”.

Given that grant income is often taken as yet another proxy for research quality, and that there’s a clear Matthew effect (rightly or wrongly) at play in science funding, this correlation between research volume and REF placement is not surprising. As the Times Higher Education article on Dorothy’s proposals went on to quote,

The government should, therefore, consider allocating block funding in proportion to the number of research-active staff at a university because that would shrink the burden on universities and reduce perverse incentives in the system, [Prof Bishop] said.

Before reacting strongly one way or another, I strongly recommend that you take the time to listen to Prof. Bishop eloquently detail her arguments in the video below.

Here’s the final slide of that presentation:

DorothyBishopRecommendations

So much rests on that final point. Ultimately, the immense time and effort devoted to/wasted on the REF boils down to a lack of trust — by government, funding bodies, and, depressingly, often university senior management — that academics cannot motivate themselves without perverse incentives like aiming for a 4* paper. That would be bad enough if we all could agree on what a 4* paper looks like…

At sixes and sevens about 3* and 4*

The post below appears in today’s Times Higher Education under the title “The REF’s star system leaves a black hole in fairness.” My original draft was improved immensely by Paul Jump‘s edits (but I am slightly miffed that my choice of title (above) was rejected by the sub-editors.) I’m posting the article here for those who don’t have a subscription to the THE. (I should note that the interview panel scenario described below actually happened. The question I asked was suggested in the interview pack supplied by the “University of True Excellence”.)


“In your field of study, Professor Aspire, just how does one distinguish a 3* from a 4* paper in the research excellence framework?”

The interviewee for a senior position at the University of True Excellence – names have been changed to protect the guilty – shuffled in his seat. I leaned slightly forward after posing the question, keen to hear his response to this perennial puzzler that has exercised some of the UK’s great and not-so-great academic minds.

He coughed. The panel – on which I was the external reviewer – waited expectantly.

“Well, a 4* paper is a 3* paper except that your mate is one of the REF panel members,” he answered.

I smiled and suppressed a giggle.

Other members of the panel were less amused. After all, the rating and ranking of academics’ outputs is serious stuff. Careers – indeed, the viability of entire departments, schools, institutes and universities – depend critically on the judgements made by peers on the REF panels.

Not only do the ratings directly influence the intangible benefits arising from the prestige of a high REF ranking, they also translate into cold, hard cash. An analysis by the University of Sheffield suggests that in my subject area, physics, the average annual value of a 3* paper for REF 2021 is likely to be roughly £4,300, whereas that of a 4* paper is £17,100. In other words, the formula for allocating “quality-related” research funding is such that a paper deemed 4* is worth four times one judged to be 3*; as for 2* (“internationally recognised”) or 1* (“nationally recognised”) papers, they are literally worthless.

We might have hoped that before divvying up more than £1 billion of public funds a year, the objectivity, reliability and robustness of the ranking process would be established beyond question. But, without wanting to cast any aspersions on the integrity of REF panels, I’ve got to admit that, from where I was sitting, Professor Aspire’s tongue-in-cheek answer regarding the difference between 3* and 4* papers seemed about as good as any – apart from, perhaps, “I don’t know”.

The solution certainly isn’t to reach for simplistic bibliometric numerology such as impact factors or SNIP indicators; anyone making that suggestion is not displaying even the level of critical thinking we expect of our undergraduates. But every academic also knows, deep in their studious soul, that peer review is far from wholly objective. Nevertheless, university senior managers – many of them practising or former academics themselves – are often all too willing, as part of their REF preparations, to credulously accept internal assessors’ star ratings at face value, with sometimes worrying consequences for the researcher in question (especially if the verdict is 2* or less).

Fortunately, my institution, the University of Nottingham, is a little more enlightened – last year it had the good sense to check the consistency of the internal verdicts on potential REF 2021 submissions via the use of independent reviewers for each paper. The results were sobering. Across seven scientific units of assessment, the level of full agreement between reviewers varied from 50 per cent to 75 per cent. In other words, in the worst cases, reviewers agreed on the star rating for no more than half of the papers they reviewed.

Granted, the vast majority of the disagreement was at the 1* level; very few pairs of reviewers were “out” by two stars, and none disagreed by more. But this is cold comfort. The REF’s credibility is based on an assumption that reviewers can quantitatively assess the quality of a paper with a precision better than one star. As our exercise shows, the effective error bar is actually ± 1*.

That would be worrying enough if there were a linear scaling of financial reward. But the problem is exacerbated dramatically by both the 4x multiplier for 4* papers and the total lack of financial reward for anything deemed to be below 3*.

The Nottingham analysis also examined the extent to which reviewers’ ratings agreed with authors’ self-scoring (let’s leave aside any disagreement between co-authors on that). The level of full agreement here was similarly patchy, varying between 47 per cent and 71 per cent. Unsurprisingly, there was an overall tendency for authors to “overscore” their papers, although underscoring was also common.

Some argue that what’s important is the aggregate REF score for a department, rather than the ratings of individual papers, because, according to the central limit theorem, any wayward ratings will “wash out” at the macro level. I disagree entirely. Individual academics across the UK continue to be coaxed and cajoled into producing 4* papers; there are even dedicated funding schemes to help them do so. And the repercussions arising from failure can be severe.

It is vital in any game of consequence that participants be able to agree when a goal has been scored or a boundary hit. Yet, in the case of research quality, there are far too many cases in which we just can’t. So the question must be asked: why are we still playing?

Concrete Reasons for the Abstract

I’ve just finished my last set of undergraduate lab report marking for this year and breathed a huge sigh of relief. Overall, however, the quality of the students’ reports has improved considerably over the year, with some producing work of a very high standard. (I get a little frustrated at times with the frustrating Daily Mail-esque whining about “students these days” that infects certain academics of a certain vintage.) Nonetheless, there remain some perennial issues with report writing…

My colleague James O’Shea sent the following missive/ cri de coeur to all of our 1st year undergrad lab class yesterday. I’m posting it here — with James’ permission, of course — because I thought it was a wonderful rationale for the importance of the abstract. (And I feel James’ pain.) Over to you, James.


 

You have written your last formal report for the first year but you will write many more in the coming years and possibly throughout your career. It seems that the purpose of abstracts and figure captions has not quite sunk in yet. This will come as you read more scientific papers (please read more scientific papers). What you want is to give a complete picture of why the experiment was needed, what the hypothesis was, how it was explored, what the result was, and what the significance of that result is. You should read your abstract back as if it is the only thing people will read. In most cases, it really is the only thing they will read. If the abstract does not provide all these things, the likely outcome is that they won’t bother reading the rest – your boss included – and all the work you put in doing the research will be for nothing.

If a researcher (or your boss) does decide – based on the abstract – that they are interested in your report or paper, they might if they are short of time first just look at the figures. The figure caption is therefore vital. Again, look at the figure and read the caption back to yourself as if this (in conjunction with the abstract) is the only thing they will read. It has to be understandable in isolation from the main body of the text. The figure represents the work that was done. The caption needs to explain that work.

If your boss did read the abstract and decided to look at the figures, they will then most likely skip to the conclusions. From this they will want to get an overview of what new knowledge now exists and what impact it will have on their company or research program. They might then recommend that others in the organisation read your report in detail to find out how robust the research is, or they might give you the go ahead to do more research, or let you lead your own team. But if your abstract did not tell the interesting story in the first place, or your figure captions did not convey what work was done, your report might not even get read in the real world.

Best regards

James O’Shea