Guilty Confessions of a REFeree

#4 of an occasional series

At the start of this week I spent a day in a room in a university somewhat north of Nottingham with a stack of research papers and a pile of grading sheets. Along with a fellow physicist from a different university (located even further north of Nottingham), I had been asked to act as an external reviewer for the department’s mock REF assessment.

I found it a deeply uncomfortable experience. My discomfort had nothing to do, of course, with our wonderfully genial hosts — thank you all for the hospitality, the conversation, the professionalism, and, of course, lunch. But I’ve vented my spleen previously on the lack of consistency in mock REF ratings (it’s been the most-viewed post at Symptoms… since I resurrected the blog in June last year) and I agreed to participate in the mock assessment so I could see for myself how the process works in practice.

Overall, I’d say that the degree of agreement on “star ratings” before moderation of my co-marker’s grading and mine was at the 70% level, give or take. This is in line with the consistency we observed at Nottingham for independent reviewers in Physics and is therefore, at least, somewhat encouraging. (Other units of assessment for Nottingham’s mock REF review had only 50% agreement.)  But what set my teeth on edge for a not-insignificant number of papers — including quite a few of those on which my gradings agreed with those of my co-marker — was that I simply did not feel at all  qualified to comment.

Even though I’m a condensed matter physicist and we were asked to assess condensed matter physics papers, I simply don’t have the necessary level of hubris to pretend that I can expertly assess any paper in any CMP sub-field. The question that went through my head repeatedly was “If I got this paper from Physical Review Letters (or Phys. Rev. B, or Nature, or Nature Comms, or Advanced Materials, or J. Phys. Chem. C…etc…) would I accept the reviewing invitation or would I decline, telling them it was out of my field of expertise?”  And for the majority of papers the answer to that question was a resounding “I’d decline the invitation.”

So if a paper I was asked to review wasn’t in my (sub-)field of expertise, how did I gauge its reception in the relevant scientific community?

I can’t quite believe I’m admitting this, given my severe misgivings about citation metrics, but, yes, I held my nose and turned to Web of Science. And citation metrics also played a role in the decisions my co-marker made, and in our moderation. This, despite the fact that we had no way of normalising those metrics to the prevailing citation culture of each sub-field, nor of ranking the quality as distinct from the impact of each paper. (One of my absolutely favourite papers of all time – a truly elegant and pioneering piece of work – has picked up a surprisingly low number of citations, as compared to much more pedestrian work in the field.)

Only when I had to face a stack of papers and grade them for myself did I realise just how exceptionally difficult it is to pass numerical judgment on a piece of work in an area that lies outside my rather small sphere of research. I was, of course, asked to comment on publications in condensed matter physics, ostensibly my area of expertise. But that’s a huge field. Not only is no-one a world-leading expert in all areas of condensed matter physics, it’s almost impossible to keep up with developments in our own narrow sub-fields of interest let alone be au fait with the state of the art in all other sub-fields.

So we therefore turn to citations to try to gauge the extent to which a paper has made ripples — or perhaps even sent shockwaves – through a sub-field in which we have no expertise. My co-marker and I are hardly alone in adopting this citation-counting strategy. But that’s of course no excuse — we were relying on exactly the type of pseudoquantitative heuristic that I have criticised in the past and I felt rather “grubby” at the end of the (rather tiring) day. David Colquhoun made the following point time and again in the run up to the last REF  (and well before):

All this shows what is obvious to everyone but bone-headed bean counters. The only way to assess the merit of a paper is to ask a selection of experts in the field.

Nothing else works.

Nothing.

Bibliometrics are a measure of visibility and “clout” in a particular (yet often nebulously defined) research community; they’re not a quantification of scientific quality. Therefore, very many scientists, and this most definitely includes me, have deep misgivings about using citations to judge a paper’s — let alone a scientist’s — worth.

Although I agree with that quote from David above, the problem is that we need to somehow choose the correct “boundary conditions” for each expert; I can have a reasonable level of expertise in one sub-area of a field — say, scanning probe microscopy or self-assembly or semiconductor surface physics — and a distinct lack of working knowledge, let alone expertise, in another sub-area of that self-same field. I could list literally hundreds of topics where I would, in fact, be winging it.

For many years, and because of my deep aversion to simplistic citation-counting and bibliometrics, I’ve been guilty of the type of not-particularly-joined-up thinking that Dorothy Bishop rightly chastises in this tweet…

We can’t trust the bibliometrics in isolation (for all the reasons (and others) that David Colquhoun lays out here), so when it comes to the REF the argument is that we have to supplement the metrics with “quality control” via another round of ostensibly expert peer review. But the problem is that it’s often not expert peer review; I was certainly not an expert in the subject areas of very many of the papers I was asked to judge. And I’ll hold that no-one can be a world-leading expert in every sub-field of a given area of physics (or any other discipline).

So what are the alternatives?

David has suggested that we should, in essence, retire what’s known as the “dual support” system for research funding (see the video embedded below): “…abolish the REF, and give the money to research councils, with precautions to prevent people being fired because their research wasn’t expensive enough.” I have quite some sympathy with that view because the common argument that the so-called QR funding awarded via the REF is used to support “unpopular” areas of research that wouldn’t necessarily be supported by the research councils is not at all compelling (to put it mildly). Universities demonstrably align their funding priorities and programmes very closely with research council strategic areas; they don’t hand out QR money for research that doesn’t fall within their latest Universal Targetified Globalised Research Themes.

Prof. Bishop has a different suggestion for revamping how QR funding is divvied up, which initially (and naively, for the reasons outlined above) I found a little unsettling. My first-hand experience earlier this week with the publication grading methodology used by the REF — albeit in a mock assessment — has made me significantly more comfortable with Dorothy’s strategy:

.”..dispense with the review of quality, and you can obtain similar outcomes by allocating funding at institutional level in relation to research volume”.

Given that grant income is often taken as yet another proxy for research quality, and that there’s a clear Matthew effect (rightly or wrongly) at play in science funding, this correlation between research volume and REF placement is not surprising. As the Times Higher Education article on Dorothy’s proposals went on to quote,

The government should, therefore, consider allocating block funding in proportion to the number of research-active staff at a university because that would shrink the burden on universities and reduce perverse incentives in the system, [Prof Bishop] said.

Before reacting strongly one way or another, I strongly recommend that you take the time to listen to Prof. Bishop eloquently detail her arguments in the video below.

Here’s the final slide of that presentation:

DorothyBishopRecommendations

So much rests on that final point. Ultimately, the immense time and effort devoted to/wasted on the REF boils down to a lack of trust — by government, funding bodies, and, depressingly, often university senior management — that academics cannot motivate themselves without perverse incentives like aiming for a 4* paper. That would be bad enough if we all could agree on what a 4* paper looks like…

Should I stay or should I go now? The postdoc mobility myth

2535963283_b42a53b173_b

First published at physicsfocus.

Far back in the mists of time, in the great and glorious days of Britpop, Forrest Gump, and John Major’s “Back to Basics”, I was a postdoc[1]. It was my first postdoctoral position after completing my PhD at Dublin City University in late 1993 and I loved the research I was doing, the working environment, and the camaraderie and teamwork of the Nottingham group.

But I hated the underlying volatility of the post.

After completing a two-year postdoc, I was funded by consecutive short-term contracts. At any time the funding stream could have dried up, and I would have had to move on. I got lucky: I secured a permanent lectureship post after three years at Nottingham and I’ve stayed there ever since – the School of Physics and Astronomy is a fantastic place to work.

I know for a fact, however, that the research ‘outputs’ I had in 1997 – enough for a lectureship at the time – wouldn’t get me within sniffing distance of a shortlist today. The bar has been raised dramatically for postdocs over the intervening years. Increasingly, the route to a permanent academic position involves first winning a fellowship through a highly competitive process.

One of the factors which is very often taken into consideration when selecting for both fellowship and lectureship positions is the “mobility” of the candidate. Indeed, the Leverhulme Trust now explicitly states in its advice to applicants that mobility is a key criterion: “Priority will be given to applicants who show evidence of mobility during their academic careers to date.”

The blunt statement that mobility will be used as a criterion in selecting fellows – with no attempt to qualify this in terms of the personal circumstances of the applicant – reveals some worryingly simplistic and out-dated thinking from the Trust. They are not alone, of course, in assuming that mobility must necessarily be an advantage for a researcher, as this recent article points out in the context of EU funding programmes. The arguments about mobility in that piece resonated with me because I coordinate a Marie Curie Initial Training Network (ITN) project which funds 14 early-career researchers across six countries. Researcher mobility for ITN projects is not only advantageous from the point of view of the European Commission, it’s essential – I can’t employ a UK national on a Marie Curie ITN contract in Nottingham. (Can someone please make sure that this nugget of EU funding policy wings its way to Nigel Farage? I want to watch him spontaneously self-combust…)

The argument that is often made – and which was voiced during a lengthy twitter debate with my Head of School[2] and others on this topic yesterday – is that a postdoc, let’s say Dr. Globetrotter, who has moved from group to group is likely to have greater drive, motivation, and scientific independence than her colleague, Dr. Stayen-Putt, who has remained at the same institution throughout her undergrad, postgrad, and postdoctoral career.

I really don’t buy this argument at all.

Skewing the selection process towards candidates who are willing to ‘up sticks’ and move to a new group every few years immediately disadvantages – and, at worst, discriminates against – those whose personal circumstances and family commitments mean that they do not have the freedom to move. I, for one, would not have been willing to disrupt my children’s lives on a regular basis simply so I could demonstrate a commitment to mobility to a fellowship panel. And I find it rather insulting that this could have been interpreted as a lack of scientific drive, motivation, and independence.

The assumption that scientific independence correlates positively with mobility also needs to be challenged. There is no evidence at all that a postdoc who has been in the same institution for their entire career is any less scientifically independent, or any less scientifically motivated, than a researcher who clocks up the air miles. Indeed, I can think of reasons why there could be a negative correlation between mobility and scientific independence – it takes considerable time to establish oneself at a new institution, to learn to interact with a new group of colleagues, and to work out how you can carve out a niche to “make a mark”.

Moreover, there’s a rather straightforward, pragmatic reason why mobility may not be conducive to establishing scientific independence. Experimental physics is not easy – the ‘kit’ is often complicated and frustratingly temperamental (particularly for non-commercial systems which the researcher has built themselves). If the experimental infrastructure in an institution is very well-matched to a researcher’s scientific goals it would be perverse for them to move simply so that they can tick the mobility box.

And finally, the wonders of the interwebs mean that researchers are connected like never before. In this context, the Leverhulme Trust’s focus on mobility as a criterion in awarding fellowships is particularly quaint, given the extent to which research groups now network and interact virtually.

Image: Britpop, an automatic association with the early 90s – but the bar has been raised for postdocs since then. Credit: Danny PiG/Flickr. Reproduced under a Creative Commons licence.

[1] …and a Douglas Adams fan.

[2] My P45 is in the post.

Selling science by the pound

mixture-69523_960_720

The President of the National Research Council (NRC) of Canada, John McDougall, caused quite a blogstorm, and set Twitter alight, at the end of last month when he said:

“Scientific discovery is not valuable unless it has commercial value.”

The tweets below give a good indication of the consensus view among the Twitterati. I don’t have a Twitter account but I also added my own small howl of outrage via The Conversation.

There’s just one small problem: McDougall didn’t say that.

As described at Phil Plait’s Bad Astronomy blog, McDougall was badly misquoted in a Toronto Sun article. This misquote effectively went ‘viral’. Shortly after my brief article at The Conversation was uploaded, I was contacted by Patrick Bookhout, Media Relations Officer at the NRC, who was understandably quite keen to put the record straight. As I told Patrick by e-mail, like too many others I took the quote at face-value. A mea culpa is in order – I didn’t spend enough time doing my homework, ie verifying that the newspaper article had got its facts straight. That I was not alone in this is no excuse.

So what did McDougall actually say? Here’s the contentious quote verbatim:

“Impact is the essence of innovation. A new idea or discovery may in fact be interesting, but it doesn’t qualify as innovation until it’s been developed into something that has commercial or societal value.”

And you know what? I agree with much of that statement. Scientific discovery and innovation are different things and, for reasons I’ll outline below, we ultimately do academic research, and the taxpayers who fund it, a disservice to pretend otherwise. (McDougall is wrong, however, in suggesting that new ideas and discoveries, in and of themselves, do not have societal value.)

Richard Jones, Pro-Vice Chancellor for Research and Innovation at the University of Sheffield, and erstwhile Strategic Advisor for Nanotechnology for the Engineering and Physical Sciences Research Council (EPSRC), has pointed out the disconnect that exists between fundamental scientific research and the ‘nucleation’ and growth of successful industries based on innovative technologies.

I’m not going to rehearse Jones’ arguments here. I would strongly recommend that you visit his Soft Machines blog for a number of extremely well-argued posts on the deficiencies in UK innovation policy. (If only all PVCs were as well-informed as Prof. Jones…). Although Richard and I may not always see eye to eye on the value of, and motivations for, basic scientific research, he is someone who certainly does his homework. Take a look at his analysis of the UK’s disinvestment in R&D since 1980. (Note, in particular, the steady decline in private sector investment and Jones’ highly plausible interpretation of what this means for the direction of academic science in the UK.)

McDougall got it right about the disparity between scientific research at the frontiers of knowledge and innovations that translate to the market. But, of course, this is not a distinction that academic scientists, and the research councils which fund them, are exactly falling over themselves to promote to government. In the short term it serves us very well indeed to blur the boundaries between funding for basic science and for near-market R&D. You reap what you sow, however, and ratcheting up expectations for short-term, and direct, returns on investment in academic research, across the board, is a rather disingenuous and dangerous strategy.

What I find truly depressing is that this strategy is now fundamentally embedded in the workings of the research councils in the UK. EPSRC, in particular, has introduced a slew of new funding mechanisms and policies over the past five years or so which are steadily ensuring that it becomes more and more difficult in the UK to get funding for disinterested research which is not connected to the near-term requirements of industry.

For the more masochistic among you, there’s much more on my ‘issues’ with EPSRC here, here, and here. (And, oh, here as well.) As I mentioned in the article in The Conversation, recent moves by EPSRC towards further skewing the funding landscape towards applied research include the recommendation that industry not only is involved in Centres for Doctoral Training, but ‘co-creates’ the PhD training programme.

Not long ago, at the 2013 Association of Research Managers and Administrators (ARMA) conference here in Nottingham, Rick Rylance, Chair of Research Councils UK, said – assuming that I can believe the Twitter traffic this time – that the distinction between pure and applied research is “beginning to become untenable”. This is a stance that is becoming increasingly fashionable and was levelled against my particular research area, nanoscience, not so long ago.

I disagree with Rylance in the strongest possible way. All scientific research indeed falls somewhere along the pure-applied spectrum, and the boundary can certainly be difficult to define. But there is a vast difference in the mindset, motivations, and working methods of an academic scientist working on, for example, the fundamental basis of quantum field theory (…or the origin of dark matter, or the location of exoplanets, or submolecular resolution imaging at 4 K …etc.), and her colleague in a nearby department who is attempting to improve the efficiency of a market-ready photovoltaic device in collaboration with industry. Germany certainly sees a distinct separation between fundamental and applied science, supporting basic science via its Max Planck Institutes, and applied research through the Fraunhofer Society.

Contrary to what Rylance states, the science funding process would be a great deal more honest and free of misleading hyperbole (directed at both government and the taxpayer) if there were a much stronger delineation of basic and applied research projects, including the provision of separate funding streams. As it stands, EPSRC’s ‘one size fits all’ approach means that, regardless of where a scientist’s work falls on the pure-applied spectrum, each and every grant proposal must outline the direct socioeconomic worth of the research via Pathways to Impact and National Importance statements.

And that’s not so very far removed from a position which holds that “Scientific discovery is not valuable unless it has commercial value.”

Image: https://pixabay.com/en/mixture-currencies-finance-business-69523/