At sixes and sevens about 3* and 4*

The post below appears in today’s Times Higher Education under the title “The REF’s star system leaves a black hole in fairness.” My original draft was improved immensely by Paul Jump‘s edits (but I am slightly miffed that my choice of title (above) was rejected by the sub-editors.) I’m posting the article here for those who don’t have a subscription to the THE. (I should note that the interview panel scenario described below actually happened. The question I asked was suggested in the interview pack supplied by the “University of True Excellence”.)


“In your field of study, Professor Aspire, just how does one distinguish a 3* from a 4* paper in the research excellence framework?”

The interviewee for a senior position at the University of True Excellence – names have been changed to protect the guilty – shuffled in his seat. I leaned slightly forward after posing the question, keen to hear his response to this perennial puzzler that has exercised some of the UK’s great and not-so-great academic minds.

He coughed. The panel – on which I was the external reviewer – waited expectantly.

“Well, a 4* paper is a 3* paper except that your mate is one of the REF panel members,” he answered.

I smiled and suppressed a giggle.

Other members of the panel were less amused. After all, the rating and ranking of academics’ outputs is serious stuff. Careers – indeed, the viability of entire departments, schools, institutes and universities – depend critically on the judgements made by peers on the REF panels.

Not only do the ratings directly influence the intangible benefits arising from the prestige of a high REF ranking, they also translate into cold, hard cash. An analysis by the University of Sheffield suggests that in my subject area, physics, the average annual value of a 3* paper for REF 2021 is likely to be roughly £4,300, whereas that of a 4* paper is £17,100. In other words, the formula for allocating “quality-related” research funding is such that a paper deemed 4* is worth four times one judged to be 3*; as for 2* (“internationally recognised”) or 1* (“nationally recognised”) papers, they are literally worthless.

We might have hoped that before divvying up more than £1 billion of public funds a year, the objectivity, reliability and robustness of the ranking process would be established beyond question. But, without wanting to cast any aspersions on the integrity of REF panels, I’ve got to admit that, from where I was sitting, Professor Aspire’s tongue-in-cheek answer regarding the difference between 3* and 4* papers seemed about as good as any – apart from, perhaps, “I don’t know”.

The solution certainly isn’t to reach for simplistic bibliometric numerology such as impact factors or SNIP indicators; anyone making that suggestion is not displaying even the level of critical thinking we expect of our undergraduates. But every academic also knows, deep in their studious soul, that peer review is far from wholly objective. Nevertheless, university senior managers – many of them practising or former academics themselves – are often all too willing, as part of their REF preparations, to credulously accept internal assessors’ star ratings at face value, with sometimes worrying consequences for the researcher in question (especially if the verdict is 2* or less).

Fortunately, my institution, the University of Nottingham, is a little more enlightened – last year it had the good sense to check the consistency of the internal verdicts on potential REF 2021 submissions via the use of independent reviewers for each paper. The results were sobering. Across seven scientific units of assessment, the level of full agreement between reviewers varied from 50 per cent to 75 per cent. In other words, in the worst cases, reviewers agreed on the star rating for no more than half of the papers they reviewed.

Granted, the vast majority of the disagreement was at the 1* level; very few pairs of reviewers were “out” by two stars, and none disagreed by more. But this is cold comfort. The REF’s credibility is based on an assumption that reviewers can quantitatively assess the quality of a paper with a precision better than one star. As our exercise shows, the effective error bar is actually ± 1*.

That would be worrying enough if there were a linear scaling of financial reward. But the problem is exacerbated dramatically by both the 4x multiplier for 4* papers and the total lack of financial reward for anything deemed to be below 3*.

The Nottingham analysis also examined the extent to which reviewers’ ratings agreed with authors’ self-scoring (let’s leave aside any disagreement between co-authors on that). The level of full agreement here was similarly patchy, varying between 47 per cent and 71 per cent. Unsurprisingly, there was an overall tendency for authors to “overscore” their papers, although underscoring was also common.

Some argue that what’s important is the aggregate REF score for a department, rather than the ratings of individual papers, because, according to the central limit theorem, any wayward ratings will “wash out” at the macro level. I disagree entirely. Individual academics across the UK continue to be coaxed and cajoled into producing 4* papers; there are even dedicated funding schemes to help them do so. And the repercussions arising from failure can be severe.

It is vital in any game of consequence that participants be able to agree when a goal has been scored or a boundary hit. Yet, in the case of research quality, there are far too many cases in which we just can’t. So the question must be asked: why are we still playing?

Concrete Reasons for the Abstract

I’ve just finished my last set of undergraduate lab report marking for this year and breathed a huge sigh of relief. Overall, however, the quality of the students’ reports has improved considerably over the year, with some producing work of a very high standard. (I get a little frustrated at times with the frustrating Daily Mail-esque whining about “students these days” that infects certain academics of a certain vintage.) Nonetheless, there remain some perennial issues with report writing…

My colleague James O’Shea sent the following missive/ cri de coeur to all of our 1st year undergrad lab class yesterday. I’m posting it here — with James’ permission, of course — because I thought it was a wonderful rationale for the importance of the abstract. (And I feel James’ pain.) Over to you, James.


 

You have written your last formal report for the first year but you will write many more in the coming years and possibly throughout your career. It seems that the purpose of abstracts and figure captions has not quite sunk in yet. This will come as you read more scientific papers (please read more scientific papers). What you want is to give a complete picture of why the experiment was needed, what the hypothesis was, how it was explored, what the result was, and what the significance of that result is. You should read your abstract back as if it is the only thing people will read. In most cases, it really is the only thing they will read. If the abstract does not provide all these things, the likely outcome is that they won’t bother reading the rest – your boss included – and all the work you put in doing the research will be for nothing.

If a researcher (or your boss) does decide – based on the abstract – that they are interested in your report or paper, they might if they are short of time first just look at the figures. The figure caption is therefore vital. Again, look at the figure and read the caption back to yourself as if this (in conjunction with the abstract) is the only thing they will read. It has to be understandable in isolation from the main body of the text. The figure represents the work that was done. The caption needs to explain that work.

If your boss did read the abstract and decided to look at the figures, they will then most likely skip to the conclusions. From this they will want to get an overview of what new knowledge now exists and what impact it will have on their company or research program. They might then recommend that others in the organisation read your report in detail to find out how robust the research is, or they might give you the go ahead to do more research, or let you lead your own team. But if your abstract did not tell the interesting story in the first place, or your figure captions did not convey what work was done, your report might not even get read in the real world.

Best regards

James O’Shea

 

 

Pressure vessels: the epidemic of poor mental health among academics

This post takes its title from a talk that will be given by Liz Morrish here at UoN next week. (5:00 pm on May 21 in The Hemsley.) Here’s the outline:

Liz Morrish will present findings that show how staff employed at Higher Education Institutions/ Universities are accessing counselling and occupational health services at an increasing rate. Between 2009 and 2015, counselling referrals have risen by 77 per cent, while staff referrals to Occupational Health services during the same period have risen by 64 per cent. This attests to an escalating epidemic of poor mental health among the sector’s employees. I will consider some of the factors which weigh on the mental health of academic staff: escalating and excessive workloads; the imposition of metric surveillance; outcomes-based performance management; increasing precarity and insecure contracts. Universities have been characterised as ‘anxiety machines’ which purposefully flout legal requirements to prevent stress in the workplace. Given the urgency of the situation, I will propose some recommendations which if institutions were to follow, might alleviate some of the pressures.

…and here’s Liz’s bio:

Liz Morrish is an independent scholar and activist for resistance to managerial appropriation of the university. She is a visiting fellow at York St John University. She was principal lecturer and subject leader of linguistics at Nottingham Trent University until speaking out and writing about the mental health of academics brought about her resignation in 2016. She is completing a co-authored book on managerial discourse in the neoliberal academy, entitled Academic Irregularities (Routledge forthcoming) and she also writes a blog with the same name: https://academicirregularities.wordpress.com/. Having exited the academy, Liz now has more time for other activities, and she now spends time as a marathon swim observer.

I met Liz a number of years ago, when she was principal lecturer at Nottingham Trent University. Not so long after we met, NTU disgracefully brought disciplinary proceedings against Liz when she spoke out about the mental health of academics, ultimately causing her to resign. For the full story on NTU’s shocking behaviour — driven, of course, by its metrics-and-league-table-infected management ‘strategy’ — an exceptionally important article written for the Times Higher Education shortly after Liz’s resignation is a must-read. Here’s a taster, but you should read the entire article for deep insights into just how low a university will go in its attempts to protect its reputation and pressure its staff:

In March last year [2016], Times Higher Education republished a blog piece that I wrote on the causes of stress and threats to mental health in academic life. The piece recounted how, on University Mental Health Day, I opened up to students about some of the pressures their lecturers were under. Many readers were kind enough to retweet the link, respond under the line or email me personally to let me know that my article resonated for colleagues around the world. But after it had received 10,000 hits on my own blog and spent four days trending on THE’s website, my previous employer objected to it and I was obliged to ask for it to be taken down. This inaugurated a disciplinary process that I felt curbed my ability to write further on the topic, or to have a frank dialogue with students on mental health in universities.

I feel very fortunate indeed that I am employed by the “other” university in Nottingham. Although I have had, and continue to have, my spats with senior management here, they have not once asked me to constrain or curtail my criticism of university (and University) culture; there’s been not so much as a quiet word in my ear following even rather scathing public critiques. Thank you, UoN, for your commitment to academic freedom.

I’d very much appreciate it if those of you who are Twitter-enabled UoN academics could spread the word about Liz’s talk. (I’ve forgone that particular form of communication.)  I hope to see you there on May 21.

 

The wit and wisdom of Associate Deans

There are very, very, very few things I miss about Twitter but the brilliantly incisive @ass_deans is certainly one…

 

The New IOP Physics Technician Award

I received an e-mail from the Institute of Physics a couple of days ago on the new IOP Technician Award and was planning to blog about it. Peter Coles beat me to it, however. His post below highlights the essential contributions of support and technical staff to universities; they are the lifeblood of everything we do. And that’s especially true for physicists of the experimental stripe like myself.

I’ve got to say that while I have the occasional moan about some aspects of my own university, Nottingham (where Peter was a colleague some time ago), when it comes to recognising the contributions of technicians, UoN has a pretty good track record. For one, it was a founding signatory of the Technician Commitment.

In the Dark

Picture Credit: Cardiff University School of Physics & Astronomy

I remember a few years ago one of my colleagues when I worked in the School of Physics & Astronomy at Cardiff University, Steven Baker, won an award for being the best STEM Technician in the category of Physical Sciences in the whole country! At the time this was a new award set up by the Higher Education Academy, so Steven was the inaugural winner of it.

Now there’s another new award, this time from the Institute of Physics and dedicated to Physics technicians (not necessarily in universities). I quote:

The IOP Technician Award enables the community to recognise and celebrate the skills and experience of technicians and their contribution to physics.

You can find full details of how to nominate an awardee here. The deadline is 14th June 2019. The prize is worth £1000, but more…

View original post 159 more words

Vying with the viva

This week’s Times Higher Education‘s cover feature is “Lighting The Way“, on the theme of PhD supervision. Along with five other academics, across a range of disciplines, I was invited by Paul Jump to contribute my thoughts on the role of the PhD supervisor. The editorial (by John Gill) sums up my central point as “the fundamental responsibility [of the PhD supervisor] is still to nurture independence such that the doctoral candidate ceases to be a student and becomes a peer.” That’s a fair summary. I also reiterated my commitment to referring to PhD researchers, rather than PhD students, in line with Jeff Ollerton’s important suggestion.

I’ve had the “Vying with the Viva” title of this post stuck in my head for a little while now and the publication of the Times Higher article seems as timely a moment as any to jot down some tips for PhD students  researchers who are preparing for a viva voce examination. I should first say that just about everything you need to know about doing a PhD is covered in a wonderful book by an alumnus of the Nottingham Nanoscience Group, James Hayton, whose PhD it was my absolute pleasure to supervise. I cannot recommend that book highly enough (and not only because it demonstrates that Dr. Hayton managed to survive my supervision and come out the other side relatively unscathed, if perhaps swearing a little more often than is strictly necessary.) James also has a great blog, website, and series of videos on the many peaks and pitfalls of doing a PhD.

I thought, however, that it might be helpful for those about to undertake a viva to hear from someone who has examined PhD candidates (as both external and internal examiner) at the rate of about three or four per year (on average) over the last couple of decades. At this point in my career, I have also been primary supervisor for a total of twenty-six students. (Twenty-two have completed their thesis to date. The remaining four are in 1st year (Oli), 2nd year (Joe), and the final year (Alex and Filipe) of their PhD project, respectively.)*

I should stress that what I write below is UK- and Ireland-centric and is from the perspective of a condensed matter physicist/nanoscientist (although I could also just about get away with calling myself a chemical physicist/physical chemist, given the research we do.) The examples chosen obviously reflect my research background and examining experience but the advice is, I would say, broadly applicable for all disciplines.

My own viva, back on a snowy January morning in 1994 — with the wonderfully-monickered Iggy McGovern, physicist and poet, as my external examiner** — was very similar in style to those I’ve since attended as examiner rather than candidate. The PhD researcher sits on one side of a table, with the examiners — one from a different university (the external), the other from the same university as the candidate — seated opposite. Sometimes (though very, very rarely in my experience), the PhD supervisor will also attend, and in Ireland it’s more common to have a moderator in place to ensure that the candidate and examiners don’t come to blows. (Joke. Usually.)

OK, on with those tips for a successful viva (in no particular order)…

1. Have a practice viva.

2. Have a practice viva.

3. Have a practice viva. Excuse my hammering home the message quite as bluntly as this but if I were forced at gunpoint to give only one piece of advice it would be the following: ask your PhD supervisor to do a mock viva with you and do not take “no” for an answer. If there’s another academic member of staff or postdoc willing to be involved, all the better — they can take the role of the internal examiner and your PhD supervisor can pretend to be the external. They should aim to grill you mercilessly. And if it takes two or three attempts at the mock viva to fully prepare you for the real thing, so be it. (The mock need only take an hour or less. That’s enough, generally, to identify where there might be issues.)

We do this in our group at Nottingham for every PhD researcher (a couple of weeks before their actual viva) and we do not hold back. They leave the mock viva feeling somewhat shell-shocked but that’s entirely the point: it’s much better to come to terms with key gaps in knowledge or understanding before the actual viva. And in the end, most of our alumni find that the real viva was a piece of cake compared to the mock.

4. Every word in your thesis is examinable.  Do not simply rearrange the words in a textbook or a review article when it comes to writing the background material. Know what those words mean. For example, if you’re an experimentalist, don’t write about particular functionals used in density functional theory (DFT) if you have absolutely no idea what a functional is (and how it differs from a function). Or, if you’re a theorist, don’t wax lyrical about phase errors in a phase-locked loop if you haven’t a clue as to how a PLL does what it does. (These are both examples I’ve encountered when I’ve been external examiner.)  And it goes without saying that you don’t cut and paste from that article or textbook. That’s plagiarism. Even if it’s just one sentence. And, no, “I couldn’t word it better than it was written” isn’t an excuse. But you know that.

5. Don’t ramble. If you don’t know the answer, just say so. Obviously, try not to reply to each question you’re asked with “I haven’t a clue” but you are not expected to know the answer to everything. Indeed, the examiner is often asking because they don’t know.  If you start rambling you can very easily start digging yourself a hole out of which it’s sometimes difficult to crawl. I certainly did this in my own viva because I chatter when I’m nervous.

6. Don’t neglect the fundamentals. This is where most PhD candidates come unstuck. There seems to be a perception that the viva will focus on the minutiae of the most arcane technical detail in your research over which you have probably lost many nights of sleep. The overwhelming odds are that your examiner won’t even have noticed this aspect of your work. They’ll focus on the much bigger picture. (See also #7.)

If your PhD is on simulating intermolecular interactions, for example, be damn sure that you are completely au fait with those pair potentials due to Morse and Lennard-Jones that you covered all the way back in Year 1 or Year 2 of your undergraduate degree. Similarly, if you’ve been determining forces from a potential energy landscape measured by an atomic force microscope, ensure that you have slightly more than a passing familiarity with scalar and vector fields. Dig out those undergrad vector calculus notes and make sure you understand how force and potential are related, for one.

You can’t, of course, prepare for every question. But it’s worth thinking carefully about which key principles of physics/chemistry underpin your research. (We’ll take the 1st and 2nd laws of thermodynamics as given. You can, of course, state the 1st and 2nd laws with confidence, right?) In the case of my viva, Prof. McGovern took me from vibration isolation for a scanning tunnelling microscope (STM), to the eddy current damping exploited in most STMs, to Faraday’s law of induction. (Thanks for that, Iggy.)

7. Think big. I tend to start the vivas I do with a simple question along the lines of “Why did you do a PhD?” or “Which aspect of your work is the most important/you’re most proud of?” or “Explain your work in a few sentences and in language that a GCSE student could understand.” My aim is to try to put the candidate at their ease. This backfires sometimes, however, because the candidate clearly is not expecting a general question of this type. Sometimes they are completely flummoxed.

A key part of the viva process is to ascertain the extent to which you understand the broader context of your work. Why is it important? Why should anyone care? What value does it have in terms of pushing your field of study forward? You need to sweat the small stuff, to borrow a phrase from our friends across the pond, but you also need to be able to see the wood for the trees.

8. “My supervisor told me to do it” is never, ever, ever the right answer. You’re being examined to assess your ability to be an independent researcher. If you don’t know why you did a particular experiment or calculation the way you did, find out right now. And ask yourself whether that really was the best way to do things. (I should note that I’ve been given “My supervisor told me to do it” as a reply on significantly more than one occasion.)

9. Forewarned is forearmed. Look up your examiners’ group web pages and publications. Take some time to familiarise yourself with the research they’ve done. Unless something has gone badly astray in the examiner selection process, their research area is not going to be light years from yours. Do your homework and you might even be able to preempt a question or two.

10. We are almost always on your side. Yes, there are one or two complete bastards out there who are deeply insecure and unpleasant individuals; they’ll take pleasure in attempting to humiliate a candidate during a viva. I’ve not encountered one of these (thus far) but I’ve certainly heard from postdocs who have had to suffer arrogant, patronising, and, in the worst cases, bullying PhD examiners.

To put this in context, however I have now done somewhere between sixty and seventy vivas (as external or internal examiner) over the course of my career to date and I’ve not encountered this type of behaviour. I would also very much hope, of course, that I have not made any of the PhD candidates I have examined feel as if they were being patronised (or worse.) We examiners want you to pass!

11. Try to enjoy yourself. Despite receiving quite a grilling from Prof. McGovern, I enjoyed my viva. It’s nerve-wracking, of course, but when you’re talking about the research you love with someone who is genuinely interested in the work, it can also be exhilarating.

No, really. It can.

I’ll leave you with a wonderfully affecting Sixty Symbols video that follows my friend and erstwhile colleague at Nottingham, James Clewett, through his viva experience…

To quote James,

“In the end…it was a very comfortable… very enjoyable experience. It was something that, in hindsight, I’d do it again.”

Oh, and that reminds me…

12Don’t wear shorts.

* Thank you Mike, Mick, Li, Rich, Fiona, Matt, Andy, Manu, James, Adam, Pete, Cong, Rosanna, Haya, Sam, Julian, Cristina, Ioannis, Morten, Jeremy, Simon, Alex, Filipe, Joe, and Oli.

** It was a lot of fun to work with Iggy seventeen years after my viva on this video:

 

Blast from the past

While searching my e-mail archive for a message from years ago, I stumbled across this unpublished submission to the letters page of the Times Higher Education. More than a decade later, I’m still smarting a little that they didn’t accept it for publication…

From: Moriarty Philip
Sent: 30 November 2008 20:48
To: letters@tsleducation.com
Subject: Comment on “‘Clever crazies quitting science” (THE 27 Nov)

Bruce Charlton of the University of Buckingham argues that modern scientists are boring because they are mild-mannered, agreeable, and socially inoffensive (News, 27 November).

What a dickhead.

Philip Moriarty, Condensed Matter Scientist

School of Physics & Astronomy
University of Nottingham
Nottingham NG7 2RD