The conference dinner chatter way of (not) correcting the scientific record

I’m reblogging this important post by Raphael Levy on the value, or lack thereof, of discussion and ‘debate’ at scientific conferences. Raphael highlights two key issues: the “behind closed doors”/”keeping it in the family” nature of scientific criticism, and, as he puts it, the play-acting that is part-and-parcel of many conference sessions. (Raphael and I, along with our colleagues at Liverpool, Nottingham, and elsewhere, spent quite some time a number of years back finding out just how much time and effort it takes to publish critique and criticism of previously published work).

Rapha-z-lab

One of the common responses of senior colleagues to my attempts to correct the scientific record goes somewhat like this:

You are giving X [leading figure in the field] too much credit anyway. We all know that there are problems with their papers. We discussed it at the latest conference with Y and Z. We just ignore this stuff and move along. Though of course X is my friend etc.

This approach is unfair, elitist and contributes to the degradation of the scientific record.

First, it is very fundamentally unfair to the many scientists who are not present at these dinner table chatters and who may believe that the accumulation of grants, prizes, and high profile papers somewhat correlate with good science. That group of scientists will include pretty much all young scientists as well as most scientists from less advantadged countries who cannot get so easily to these conferences…

View original post 179 more words

ECR blues: Am I part of the problem?

A very quick lunchtime post to highlight that this week’s Nature is a special issue on the theme of young scientists’ careers, and, as it says loud and clear on the front cover, their struggle to survive in academia. There are a number of important and timely articles on just how tough it is for early career researchers (the ECRs of the title of this post), including a worrying piece by Kendall Powell: “Young, Talented and Fed-Up“.

One of the things that struck me in the various statistics and stories presented by Nature is the following graph:

AgingWorkforce.png

Note how older scientists (and I’m soundly in the 41-55 bracket) now hold the large majority of NIH grants, and how different it was back in 1980. I’d like to know the equivalent distribution for grants in physics. If anyone can point me (in the comments section) towards appropriate statistics, I’d appreciate it.

In any case, I recommend taking a read of those articles in this week’s Nature, regardless of where you happen to be on the academic career ladder. As Powell’s article points out, Nature got a short, sharp response to its tweeted question about the challenges facing ECRs…

Politics. Perception. Philosophy. And Physics.

Today is the start of the new academic year at the University of Nottingham (UoN) and, as ever, it crept up on me and then leapt out with a fulsome “Gotcha”. Summer flies by so very quickly. I’ll be meeting my new 1st year tutees this afternoon to sort out when we’re going to have tutorials and, of course, to get to know them. One of the great things about the academic life is watching tutees progress over the course of their degree from that first “getting to know each other” meeting to when they graduate.

The UoN has introduced a considerable number of changes to the “student experience” of late via its Project Transform process. I’ve vented my spleen about this previously but it’s a subject to which I’ll be returning in the coming weeks because Transform says an awful lot about the state of modern universities.

For now, I’m preparing for a module entitled “The Politics, Perception and Philosophy of Physics” (F34PPP) that I run in the autumn semester. This is a somewhat untraditional physics module because, for one thing, it’s almost entirely devoid of mathematics. I thoroughly enjoy  F34PPP each year (despite this amathematical heresy) because of the engagement and enthusiasm of the students. The module is very much based on their contributions — I am more of a mediator than a lecturer.

STEM students are sometimes criticised (usually by Simon Jenkins) for having poorly developed communication skills. This is an especially irritating stereotype in the context of the PPP module, where I have been deeply impressed by the quality of the writing the students submit. As I discuss in the video below (an  overview of the module), I’m not alone in recognising this: articles submitted as F34PPP coursework have been published in Physics World, the flagship magazine of the Institute of Physics.

 

In the video I note that my intention is to upload a weekly video for each session of the module. I’m going to do my utmost to keep this promise and, moreover, to accompany each of those videos with a short(ish) blog post. (But, to cover my back, I’ll just note in advance that the best laid schemes gang aft agley…)

Addicted to the brand: The hypocrisy of a publishing academic

Back in December I gave a talk at the Power, Acceleration and Metrics in Academic Life conference in Prague, which was organised by Filip Vostal and Mark Carrigan. The LSE Impact blog is publishing a series of posts from those of us who spoke at the conference. They uploaded my post this morning. Here it is…


 

I’m going to put this as bluntly as I can; it’s been niggling and nagging at me for quite a while and it’s about time I got it off my chest. When it comes to publishing research, I have to come clean: I’m a hypocrite. I spend quite some time railing about the deficiencies in the traditional publishing system, and all the while I’m bolstering that self-same system by my selection of the “appropriate” journals to target.

Despite bemoaning the statistical illiteracy of academia’s reliance on nonsensical metrics like impact factors, and despite regularly venting my spleen during talks at conferences about the too-slow evolution of academic publishing towards a more open and honest system, I nonetheless continue to contribute to the problem. (And I take little comfort in knowing that I’m not alone in this.)

One of those spleen-venting conferences was a fascinating and important event held in Prague back in December, organized by Filip Vostal and Mark Carrigan: “Power, Acceleration, and Metrics in Academic Life”. My presentation, The Power, Perils and Pitfalls of Peer Review in Public – please excuse thePartridgian overkill on the alliteration – largely focused on the question of post-publication peer review (PPPR) via online channels such as PubPeer. I’ve written at length, however, on PPPR previously (here,here, and here) so I’m not going to rehearse and rehash those arguments. I instead want to explain just why I levelled the accusation of hypocrisy and why I am far from confident that we’ll see a meaningful revolution in academic publishing any time soon.

Let’s start with a few ‘axioms’/principles that, while perhaps not being entirely self-evident in each case, could at least be said to represent some sort of consensus among academics:

  • A journal’s impact factor (JIF) is clearly not a good indicator of the quality of a paper published in that journal. The JIF has been skewered many, many times with some of the more memorable and important critiques coming from Stephen Curry, Dorothy Bishop, David Colquhoun, Jenny Rohn, and, most recently, this illuminating post from Stuart Cantrill. Yet its very strong influence tenaciously persists and pervades academia. I regularly receive CVs from potential postdocs where they ‘helpfully’ highlight the JIF for each of the papers in their list of publications. Indeed, some go so far as to rank their publications on the basis of the JIF.
  • Given that the majority of research is publicly funded, it is important to ensure that open access publication becomes the norm. This one is arguably rather more contentious and there are clear differences in the appreciation of open access (OA) publishing between disciplines, with the arts and humanities arguably being rather less welcoming of OA than the sciences. Nonetheless, the key importance of OA has laudably been recognized by Research Councils UK (RCUK) and all researchers funded by any of the seven UK research councils are mandated to make their papers available via either a green or gold OA route (with the gold OA route, seen by many as a sop to the publishing industry, often being prohibitively expensive).

With these three “axioms” in place, it now seems rather straight-forward to make a decision as to the journal(s) our research group should choose as the appropriate forum for our work. We should put aside any consideration of impact factor and aim to select those journals which eschew the traditional for-(large)-profit publishing model and provide cost-effective open access publication, right?

Indeed, we’re particularly fortunate because there’s an exemplar of open access publishing in our research area: The Beilstein Journal of Nanotechnology. Not only are papers in the Beilstein J. Nanotech free to the reader (and easy to locate and download online), but publishing there is free: no exorbitant gold OA costs nor, indeed, any type of charge to the author(s) for publication. (The Beilstein Foundation has very deep pockets and laudably shoulders all of the costs).

But take a look at our list of publications — although we indeed publish in the Beilstein J. Nanotech., the number of our papers appearing there can be counted on the fingers of (less than) one hand. So, while I espouse the three principles listed above, I hypocritically don’t practice what I preach. What’s my excuse?

In academia, journal brand is everything. I have sat in many committees, read many CVs, and participated in many discussions where candidates for a postdoctoral position, a fellowship, or other roles at various rungs of the academic career ladder have been compared. And very often, the committee members will say something along the lines of “Well, Candidate X has got much better publications than Candidate Y”…without ever having read the papers of either candidate. The judgment of quality is lazily “outsourced” to the brand-name of the journal. If it’s in a Nature journal, it’s obviously of higher quality than something published in one of those, ahem, “lesser” journals.

If, as principal investigator, I were to advise the PhD students and postdocs in the group here at Nottingham that, in line with the three principles above, they should publish all of their work in the Beilstein J. Nanotech., it would be career suicide for them. To hammer this point home, here’s the advice from one referee of a paper we recently submitted:

“I recommend re-submission of the manuscript to the Beilstein Journal of Nanotechnology, where works of similar quality can be found. The work is definitively well below the standards of [Journal Name].”

There is very clearly a well-established hierarchy here. Journal ‘branding’, and, worse, journal impact factor, remain exceptionally important in (falsely) establishing the perceived quality of a piece of research, despite many efforts to counter this perception, including, most notably, DORA. My hypocritical approach to publishing research stems directly from this perception. I know that if I want the researchers in my group to stand a chance of competing with their peers, we have to target “those” journals. The same is true for all the other PIs out there. While we all complain bitterly about the impact factor monkey on our back, we’re locked into the addiction to journal brand.

And it’s very difficult to see how to break the cycle…

We are anonymous. We are legion. We are (mostly) harmful.

This revelation appeared in my Twitter timeline earlier this week:

On the same day, Nature News published a fascinating interview with Brandon Stell, the founder of PubPeer who has revealed his identity to the world:

I’ve waxed lyrical about PubPeer a number of times before, going so far as to say that its post-publication peer review (PPPR) ethos has to be the future of scientific publishing. (I now also try to include mention of PubPeer in every conference presentation/seminar I give). I’ll be gutted if PPPR of the type pioneered by PubPeer does not become de rigueur for the next generation of scientists; our conventional peer review system is, from so many perspectives, archaic and outdated. I agree entirely with Stell’s comments in that interview for Nature News:

Post-publication peer review has the potential to completely change the way that science is conducted. I think PubPeer could help us to move towards an ideal scenario where we can immediately disseminate our findings and set up a different way of evaluating significant research than the current system.

But one major bone of contention that I’ve always had with PubPeer’s approach, and that has been the subject of a couple of amicable ‘tweet-spats’ with the @PubPeer Twitter feed, is the issue of anonymity. I was disappointed that not only was it just Stell who revealed his identity — his two PubPeer co-founders remain anonymous — but that there are plans (or at least aspirations) to “shore up”, as Stell puts it, the anonymity of PubPeer commenters.

I am not a fan of internet anonymity. At all. I understand entirely the arguments regularly made by PubPeer (and many others) in favour of anonymous commenting. In particular, I am intensely aware of the major power imbalance that exists between, for example, a 1st year PhD student commenting on a paper at PubPeer and the world-leading, award-winning, scientifically decorated and oh-so-prestigious scientist whose group carried out the work that is being critiqued/attacked. Similarly, and in common with Peter Coles, I personally know bloggers who write important, challenging, and influential posts while remaining anonymous.

I also fully realise that there are are extreme cases when it might not only be career-theatening, but life-threatening for a blogger to reveal their identity. However, those are exactly that: extreme cases. It’s statistically rather improbable that all of those pseudonymously venting their spleen under articles at, say, The Guardian, Telegraph, or, forgive me, Daily Mail website are writing in fear of their life. (Although, and as this wonderful Twitter account highlights so well, many Daily Mail readers certainly feel as if their entire culture, identity, and belief system are under constant attack from the ranked hordes of migrants/PC lefties/benefit claimants/gypsies/BBC executives [delete as appropriate] swamping the country).

I am firmly of the opinion that the advantages of anonymity are far outweighed by the difficulties associated with fostering an online culture where comments and critique — and, at worst, vicious abuse — are posted under cover of a pseudonym. For one thing, there’s the strong possibility of sockpuppets being exploited to distort debate. Julian Stirling, an alumnus of the Nanoscience Group here at Nottingham and now a research fellow at NIST, has described the irritations and frustrations of the sockpuppetry we’ve experienced as part of our critique of a series of papers spanning a decade’s worth of research. In the later stages of this tussle, the line separating sockpuppetry from outright identity theft was crossed. You might suggest, like PubPeer, that this type of behaviour is not the norm. Perhaps. But our experience shows just how bad it can get.

Even in the absence of sockpuppetry, I’ve got to come clean and admit that I’m really not entirely comfortable with communicating with someone online who is not willing to reveal their identity. I’ve been trying to get to grips with just what it is about anonymous/ pseudonymous comments that rankles with me so much, having been involved in quite a number of online ‘debates’ where the vast majority of those commenting have used pseudonyms. When challenged on the use of a pseudonym, and asked for some information about their background, the response is generally aggressively defensive. Their standard rebuttal is to ask why I should care about who they are because isn’t it the strength of the argument, not the identity of the person making the comments, that really matters?

In principle, yes. But the online world is often not very principled.

Ultimately, I think that my deep irritation with pseudonyms stems from two key factors. The first of these is the fundamental lack of fairness due to the ‘asymmetry’ in communication. Dorothy Bishop wrote a characteristically considered, thoughtful, and thought-provoking piece on this issue of communication asymmetry a number of years back (in the context of the debate regarding the burqa ban). The asymmetry that’s established via anonymous commenting means that those who are critiquing an author’s (or a group’s) work are free to comment with impunity; they can say whatever they like in the clear knowledge that there’s a negligible chance of their comments ever being traced back to them.

Stell and his PubPeer co-founders claim that this is actually a key advantage of anonymity — those who comment are not constrained by concerns that they’ll be identified. But if their arguments are sound, and expressed in a polite, if critical, manner, then why the heck should they be concerned?  After all, it’s long been the case that “old school” journals — the APS’ Physical Review family of titles being a notable example — publish critiques of papers that have previously appeared in their pages. Those formal critiques are published with the names of the authors listed for all the world, or at least the readership of the journal, to see.

We should aim to change the culture so that critiquing other scientists’ work is seen as part-and-parcel of the scientific process, i.e. something for which researchers, at any career level, should be proud to take credit. Instead, the ease of commenting online from behind cover of a pseudonym or avatar is encouraging a secretive, and, let’s be honest, a rather grubby, approach to scientific criticism. I was therefore particularly encouraged by this announcement from The Winnower yesterday. It’s a fantastic idea to publish reviews of papers from journal club discussions and it’ll help to move the critique of published science to a rather more open, and thus much healthier, place.

The second, although closely related, aspect of anonymity that winds me up is that it essentially (further) depersonalises online communication. This helps to normalise a culture in which those commenting don’t ever take responsibility for what they say, or, in the worst cases — if we consider online communication in a broader context than just the scientific community — fail to appreciate just how hurtful their abuse might be. I’ve often seen comments along the lines of “It’s all just pixels on a screen. They should toughen up”. These comments are invariably made by those hiding behind a pseudonym.

As ever, xkcd has the perfect riposte…

This splendid poem also makes the point rather well. “Cuts and bruises now have healed, it’s words that I remember.

Anonymity contributes to a basic lack of online respect and too often can represent a lack of intellectual courage. When we criticise, critique, lambaste, or vilify others online let’s have the courage of our convictions and put our name to our comments.