Perplexed by Pauli

Wolfgang_PauliOriginally published at physicsfocus.

A rather pervasive meme claiming that nothing ever really touches anything else has been circulating on the internet for a number of years. I think, although I’m not entirely certain, that it may well have its origins in an explanation by a certain Michio Kaku. This type of explanation later formed the basis of a video, You Can’t Touch Anything, from the immensely successful VSauce YouTube channel, which has now accrued nearly 3.4 million views.

I appreciate just how difficult it is to explain complicated physics for a general audience (see, for example, this article in the education issue of Physics World published earlier this year). And I also fully understand that we all goof at times – particularly, and especially, me. But Kaku has got form when it comes to over-simplifying explanations to the point of incorrectness in order to exploit the ‘Wow! Quantum! Physics!’ factor. This misleading over-simplification is similarly a hallmark of the ‘you can’t touch anything’ meme.

Why is it that this particular meme winds me up so much? (After all, there’s a universe of other, much more egregious, stuff on the internet to worry about.) Well, I think it’s mainly because it hits just a little too close to home. My research area is known as non-contact atomic force microscopy (NC-AFM) and there’s a very good reason indeed why scientists in the field draw a distinction between the non-contact and contact modes of AFM. I’ve banged on about the flaws in the meme, as I see them, to Brady Haran on a number of occasions over the last couple of years and this finally led to a video, uploaded to the Sixty Symbols channel last week, where he and I debate whether atoms touch.

If you’ll excuse the shameless self-promotion1, of all the Sixty Symbols videos I’ve done with Brady, I’m most happy with this one. It shows science as a debate with evidence, models, and analogies being thrown into the mix to support a particular viewpoint – not as something which is “done and dusted” by the experts and passed down as received wisdom to the ‘masses’. This is exactly how science should work and how it should be seen to work. (Here’s the obligatory supporting Feynman quote: “Science is the belief in the ignorance of experts”. Stay tuned – another Feynman quote will be along soon.)

The reason I’m writing this post, however, isn’t to rake over the ashes of the debate with Brady (and the associated lengthy comments thread under the video). It’s instead to address a big, and quite deliberate, gap in the video: just how does the Pauli exclusion principle affect how atoms interact/touch/bond/connect? This is an absolutely fascinating topic that not only has been the subject of a vast amount of debate and confusion over many decades, but, as we’ll see, fundamentally underpins the latest developments in sub-molecular resolution AFM.

Beyond atomic resolution

At about the same time as the Sixty Symbols video was uploaded, and entirely coincidentally, a book chapter my colleagues and I have been working on over the last couple of months appeared on the condensed matter arXiv: Pauli’s Principle in Probe Microscopy. The Pauli exclusion principle (PEP) plays an essential role in state-of-the-art scanning probe microscopy, where images like that shown on the right below are increasingly becoming the norm. Scanning probe images of this type are captured by measuring the shift in resonant frequency of a tiny tuning fork to which an atomically (or molecularly) sharp tip is attached. As the probe is moved back and forth on nanometre and sub-nanometre length-scales, the gradient of the force between the tip apex and the molecule changes and this causes a change in the resonant frequency of the tuning fork. These shifts in frequency can be converted to an image or mathematically inverted to determine the tip-sample force. Or they can be listened to.

NTCDIThe image shown above is from recent work by our group at Nottingham but I really must name-check the researchers who pioneered this type of ultrahigh resolution imaging: Leo Gross and co-workers at IBM Zurich. Leo and his colleagues first demonstrated that it is possible to acquire AFM images of molecules where the entire chemical architecture can be visualised. The images show a remarkable, and almost eerie, similarity to the textbook ball-and-stick molecular models so familiar to any scientist. Compare the experimental image of NTCDI molecules on the right above to the ball-and-stick diagram on the left where grey, blue, and red spheres represent carbon, nitrogen, and oxygen atoms respectively. These exceptionally detailed images of molecular structure2 are acquired by exploiting the repulsion of electrons due to the Pauli exclusion force at very small tip-sample separations.

I was explaining all of this to a class of first-year undergraduate students last year, stressing that the repulsion we observe at small tip-sample separations – and, indeed, the repulsion ultimately responsible for the reaction force keeping them from falling through their seats – is not simply due to electrostatic repulsion of ‘like’ charges. I wound up the lecture, chucking in the throwaway remark, “…of course, the force due to Pauli exclusion isn’t really a force like any other. You’ll cover this in quantum statistics next year.”

By the time I’d got back to my office, two email messages from students in the lecture had already made their way to my inbox: “If it isn’t a force like any other, then what the heck is it?”

That’ll teach me to be flippant with the first-years. It’s a great question. Where does the repulsive force due to Pauli exclusion come from – just why is it that electrons don’t want to be ‘squeezed’ into the same quantum state?

The Quantum Identity Crisis

Ultimately, the Pauli exclusion principle has its origins in the indistinguishability of electrons. (Well, OK, fermions – but let’s stick with the PEP in the context of force microscopy.) One frustrating aspect of the discussions of quantum statistics in various textbooks, however, is that the terms ‘identical’ and ‘indistinguishable’ are too often assumed to be synonymous. Electrons are certainly identical in the sense that their ‘internal’ properties such as mass and charge are the same, but are they really indistinguishable?

Fleicschhauer had this to say in an fascinating commentary published a few years ago:

“In the quantum world, particles of the same kind are indistinguishable: the wavefunction that describes them is a superposition of every single particle of that kind occupying every allowed state. Strictly speaking, this means that we can’t talk, for instance, about an electron on Earth without mentioning all the electrons on the Moon in the same breath.”

Well, in principle, yes, we should consider the entire multi-particle ‘universal’ wavefunction. But I’m a dyed-in-the-wool, long-of-tooth and grizzled-of-beard experimentalist. I want to see evidence of this universal coupling. And you know what? As hard as I might look, I’m never going to find any experimental evidence that an electron on the Moon has any role at all to play in a force-microscopy experiment (or a chemical reaction, or an intra-atomic transition, or…) involving electrons on Earth.

I’ll stress again that in principle, the electrons are indeed indistinguishable as there is always some finite wavefunction overlap, because there is no such thing as the infinite potential well which is the mainstay of introductory quantum physics courses. In this sense, an electron on Earth and an electron on the Moon (or on Alpha Centauri) are indeed ‘coupled’ to some degree and arguably ‘indistinguishable’. But the degree of wavefunction coupling and associated energy splitting are so incredibly tiny and utterly negligible – if I can be forgiven the understatement – that, in any practical sense, the electrons are completely distinguishable.

(Some of you might at this point be having a déjà vu moment. This is possibly connected to the (over-)heated debate that stemmed from Brian Cox’s discussion of the exclusion principle in a BBC programme a few years ago. Brian caught a lot of online flak for his explanation, some of it rather too rant-y and intemperate in tone – even for me. One of the best analyses of the furore out there is a pithy blog post by Jon Butterworth for the Guardian – very well worth a read. My colleagues at Nottingham have also discussed the controversy, and Telescoper asked if Brian had Cox-ed up the explanation of the exclusion principle.)

It is only when there is appreciable wavefunction overlap, as when the atom at the very end of the AFM tip is moved very close to a molecule underneath it (or, equivalently, in a chemical bond), that the PEP ‘kicks in’ in any appreciable way. If you want to know just why indistinguishability and the exclusion principle are so intimately connected, and how electron spin plays into all of this, I’m afraid I’m going to have to refer you to Section 1.3 of that book chapter and references therein. If you’re willing to take my word for it for now, however, read on.

Fourier and Force

Let’s cut to the chase and elucidate where that repulsive ‘Pauli force’ comes from. The PEP tells us that we can’t have two electrons with the same four quantum numbers, i.e. we can’t ‘push’ them into the same quantum state. But just how does this give rise to a force between two electrons that is beyond their ‘natural’ electrostatic repulsion? Let’s strip the problem right down to its bare bones and consider a simple gedanken experiment.

Take two electrons of the same fixed spin separated by a considerable distance from each other. We’re going to move those electrons together until their wavefunctions overlap. As the electrons get closer their wavefunctions effectively change shape so that the mutual overlap is minimal – this is the PEP in action. The figure below, adapted from a description of the origin of the exclusion principle by Julian Su, schematically illustrates this effect.

FourierForce

On the left hand side the electron wavefunctions are not constrained by the exclusion principle, while on the right the PEP has been ‘switched on’. The essential point is this: the exclusion principle causes wavefunction curvature to increase. Because the kinetic energy (KE) of an electron is directly proportional to wavefunction curvature via the KE operator in quantum mechanics, increased curvature means increased kinetic energy. Or – and this is the description I much prefer because it’s yet another example of the beauty and elegance of Fourier transforms – higher wavefunction curvature requires the introduction of higher spatial frequency (i.e. higher momentum) contributions in Fourier space. It this change in the momentum distribution which gives rise to the Pauli repulsion force.

While this captures some of the essence of the exclusion principle (and certainly is enough to provide important insights into what’s going on in force microscopy experiments), it doesn’t even begin to scratch the surface of the underlying physics. I suspect that Pauli himself would dismiss all of the above with his trademark “… es ist nicht einmal falsch”. He himself said in his Nobel prize lecture of 1946 that “I was unable to give a logical reason for the Exclusion Principle or to deduce it from more general assumptions…” Almost two decades later, Feynman had the following to say:

“It appears to be one of the few places in physics where there is a rule which can be stated very simply, but for which no one has found a simple and easy explanation. The explanation is deep down in relativistic quantum mechanics. This probably means that we do not have a complete understanding of the fundamental principle involved.”

Like Feynman, I remain somewhat perplexed by Pauli’s principle.

 

_ _ _

1 In these social-media-enabled times, I guess that shameless self-promotion has become the academic’s stock-in-trade. Perhaps, however, it was ever thus.

2 There has, however, been a great deal of controversy of late as to the origin of the intermolecular features observed in AFM images by a number of groups, including ourselves. See our chapter on the role of the exclusion principle in probe microscopy for more detail.

 

Lies, damned lies, and Ofsted’s pseudostatistics

networking.jpg

First published at physicsfocus.

It’s been a week since Michael Gove was unceremoniously given the boot from his role as Education Secretary. The cheers of teachers still echo around staff rooms and schoolyards up and down the country.

Gove was variously described as incredibly unpopular, a hate figure, utterly ruthless, and a “toxic liability”. And that was just by his colleagues in the Coalition. (Allegedly.) Those who shared his simple-minded, wilfully uninformed, and proto-Victorian views on education, including a certain Richard Littlejohn, saw Gove’s unpopularity as arising simply because he was driving through what they considered to be essential reforms of an ailing education system. (My deep apologies for the preceding link to a Daily Mail article and its associated sidebar of shame. It won’t happen again. I also offer a posthumous apology to those Victorians who would likely have baulked at the suggestion that their educational methods were as backward-looking as those of Gove.)

Just why are Littlejohn and his reactionary ilk so certain that the English education system is, as they’d have it, going to hell in a handcart? A very large part of the reason is that they naively, quaintly, yet dangerously assume that education is equivalent to a competitive sport where schools, teachers, and children can be accurately assessed on the basis of positions in league tables. What’s worse – and this is particularly painful for a physicist or, indeed, anyone with a passing level of numeracy, to realise – is that this misplaced and unscientific faith in the value of statistically dubious inter-school comparisons is at the very core of the assessment culture of the Office for Standards in Education, Children’s Services and Skills (Ofsted).

An intriguing aspect of the swansong of Gove’s career as Education Secretary was that he more than once ‘butted heads’ with Michael Wilshaw, head of Ofsted. One might perhaps assume that this was a particularly apposite example of “the enemy of mine enemy is my friend”. Unfortunately not. Ofsted’s entirely flawed approach to the assessment of schools is in many ways an even bigger problem than Gove’s misplaced attempts to rewind education to the halcyon, but apocryphal, days of yore.

Moreover, Gove’s gone. Ofsted is not going anywhere any time soon.

I’ve always been uncomfortable about the extent to which number-abuse and pseudostatistics might be underpinning Ofsted’s school assessment procedures. But it was only when I became a parent governor for my children’s primary school, Middleton Primary and Nursery School in Nottingham, that the shocking extent of the statistical innumeracy at the heart of Ofsted’s processes became clear. (I should stress at this point that the opinions about Ofsted expressed below are mine, and mine alone.)

Middleton is a fantastic school, full of committed and inspirational teachers. But, like the vast majority of schools in the country, it is subject to Ofsted’s assessment and inspection regime. Ofsted’s implicit assumption is that the value of a school like Middleton, and, by extension, the value of the teachers and students in that school, can be reduced to a set of objective and robust ‘metrics’ which can in turn be used to produce a quantitative ranking (i.e. a league table). Even physicists, who spend their career wading through reams of numerical data, know full well that not everything that counts can be counted. (By the way, I use the adjective “inspirational” unashamedly. And because it winds the likes of Littlejohn and Toby Young up. As, I’d imagine, does starting a sentence with a conjunction and ending it with a preposition.)

But let’s leave the intangible and unquantifiable aspects of a school’s teaching to one side and instead critically consider the extent to which Ofsted’s data and processes are, to use that cliché beloved of government ministers, fit for purpose. In its advice to governors, Ofsted – rather ironically, as we’ll see — stresses the key importance of objective data and highlights that the governing board should assess the school’s performance on the basis of a number of measures which are ‘helpfully’ summarised at websites such as the Ofsted Data Dashboard and RAISE Online.

Ofsted’s advice to governors tacitly assumes that the data it provides, and the overall assessment methodology which gives rise to those data, are objective and can be used to robustly monitor the performance of a given school against others. Let’s just take a look at the objective evidence for this claim.

During the governor training sessions I attended, I repeatedly asked to what extent the results of Ofsted inspections (and other Ofsted-driven assessment schemes) were reproducible. In other words, if we repeated the inspection with a different set of inspectors, would we get the same result? If not, in what sense could Ofsted claim that the results of an inspection were objective and robust? As you might perhaps expect, I singularly failed to get a particularly compelling response to this question. This was for a very good reason: the results of Ofsted inspections are entirely irreproducible. A headline from the Telegraph in March this year said it all: Ofsted inspections: You’d be better off flipping a coin. This was not simply media spin. The think-tank report, “Watching the Watchmen”, on which the article was based, actually goes further: “In fact, overall the results are worse than flipping a coin”.

It’s safe to say that the think-tank in question, Policy Exchange, is on the right of the political spectrum. It is also perhaps not entirely coincidental that one of its founding members was a certain Michael Gove, and that the Policy Exchange report on Ofsted was highlighted by the right-of-centre press during the period of spats between Wilshaw and Gove mentioned above. None of that, however, detracts from the data cited in the report. These resulted from the work of Robert Coe and colleagues at Durham University and stemmed from a detailed study involving more than 3000 teachers. Coe has previously criticised Ofsted’s assessment methods in the strongest possible terms, arguing that they are not “research-based or evidence-based”.

Ofsted asks governors to treat its data as objective and make conclusions accordingly. However, without a suitable ‘control’ study – which in this case is as simple as running independent assessments of the same class with different inspectors – the data on inspections simply cannot be treated as objective and reliable. In this sense, Ofsted is giving governors, schools, and, more generally, the public exceptionally misleading messages.

But it gets worse…

The lack of rigour in Ofsted’s inspections is just one part of the problem. It’s compounded in a very worrying way by the shocking abuse of statistics that forms the basis of the Data Dashboard and RAISE Online. Governors are presented with tables of data from these websites and asked to make ‘informed’ decisions on the basis of the numbers therein. This, to be blunt, is a joke.

It would take a lengthy series of blog posts to highlight the very many flaws in Ofsted’s approach to primary and secondary school data. Fortunately, those posts have already been written by a teacher who has to deal with Ofsted’s nonsense on what amounts to a daily basis. I thoroughly recommend that you head over to the Icing On The Cake blog where you’ll find this, this, and this. The latter post is particularly physicist-friendly, given that it invokes Richard Feynman’s “cargo cult science” description of pseudoscientific methods (in the context of Ofsted’s methodology). It’s also worth following Icing On The Cake on Twitter if you’d like regular insights into the level of the data abuse which teachers have to tolerate from Ofsted.

Coincidentally, I stumbled across that blog after I had face-palmed my way (sometimes literally) through a meeting in which the Ofsted Data Dashboard tables were given to governors. I couldn’t quite believe that Ofsted presented the data in a way such that the average first-year physics or maths undergraduate could drive a horse and carriages right through it (if you’ll excuse the Goveian metaphor). So I went home and googled the simple term “Ofsted nonsense”. Right at the top of the list of hits were the Icing On The Cake posts (followed by links to many other illuminating analyses of Ofsted’s assessment practices).

I’m not going to rehash those posts here – if you’ve got even a passing interest in the education system in England you should read them (and the associated comments threads) for yourself and reach your own conclusions. To summarise, the problems are multi-faceted but can generally be traced to simple “rookie” flaws in data analysis. These include:

  1. Inadequate appreciation of the effects of small sample size;
  2. A lack of consideration of statistical significance/uncertainties in the data. (Or, at best, major deficiencies in communicating and highlighting those uncertainties);
  3. Comparison of variations between schools when the variation within a given school (from year to year) can be at least as large;
  4. An entirely misleading placement of schools in “quintiles” when the difference between the upper and lower quintiles can be marginal. Ofsted has already had to admit to a major flaw in its initial assignment of quintiles.

What is perhaps most galling is that many A-level students in English schools will be taught to recognise and avoid these types of pitfall in data analysis. It is an irony too far that those teaching the correct approach to statistics in English classrooms are assessed and compared to their peers on the basis of Ofsted’s pseudostatistical nonsense.

Image: Manipulating data. Credit: https://www.publicdomainpictures.net/en/view-image.php?image=45242

How universities incentivise academics to short-change the public

Euro Money Coins Loose Change Specie CurrencyThis is going to be a short post (for a change). First, you should read this by David Colquhoun. I’ll wait until you get back. (You should sign the petition as well while you’re over there).

In his usual down-to-earth and incisive style, Colquhoun has said just about everything that needs to be said about the shocking mismanagement of King’s College London.

So why am I writing this post? Well, it’s because KCL is far from alone in using annual grant income as a metric for staff assessment – the practice is rife across the UK higher education sector. For example, the guidance for performance review at Nottingham contains this as one of the assessment standards: “Sustained research income equal to/in excess of Russell Group average for the discipline group”. Nottingham is not going out on a limb here – our Russell Group ‘competitors’ have similar aspirations for their staff.

What’s wrong with that you might ask? Surely it’s your job as an academic to secure research income?

No. My job as an academic is to do high-quality research. Not to ‘secure research income’. It’s all too easy to forget this, particularly as a new lecturer when you’re trying to get a research group established and gain a foothold on the career ladder. (And as a less-new lecturer attempting to tick the boxes for promotion. And as a grizzled old academic aiming to establish ‘critical mass’ on the national or international research ‘stage’.)

What’s particularly galling, however, is that the annual grant income metric is not normalised to any measure of productivity or quality. So it says nothing about value for money. Time and time again we’re told by the Coalition that in these times of economic austerity, the public sector will have to “do more with less”. That we must maximise efficiency. And yet academics are driven by university management to maximise the amount of funding they can secure from the public pot.

Cost effectiveness doesn’t enter the equation. Literally.

Consider this. A lecturer recently appointed to a UK physics department, Dr. Frugal, secures a modest grant from the Engineering and Physical Sciences Research Council for, say, £200k. She works hard for three years with a sole PhD student and publishes two outstanding papers that revolutionise her field.

Her colleague down the corridor, Prof. Cash, secures a grant for £4M and publishes two solid, but rather less outstanding, papers.

Who is the more cost-effective? Which research project represents better value for money for the taxpayer?

…and which academic will be under greater pressure from management to secure more research income from the public purse?

Image: Coins, the acquistion of which is not university departments’ main aim. Credit: https://www.maxpixel.net/Golden-Gold-Riches-Treasure-Rich-Coins-Bounty-1637722

The laws of physics are undemocratic

shard-1841278_960_720

Yesterday saw the start of the Circling the Square conference at the University of Nottingham. This is a rather unusual meeting which has the lofty aim of bringing together social scientists, those in the arts and humanities, policy ‘wonks’ (for want of a better term), science communicators, and natural scientists (including physicists, of course) to discuss the various interconnected aspects of research, politics, media, and impact.

As one of the conference organisers, I was delighted that the first day featured fascinating keynote lectures, lively discussion, and a rather heated exchange amongst panellists (more on this below). In the afternoon, two of the UK’s most successful science bloggers, David Colquhoun and physicsfocus’s own Athene Donald, gave their thoughts and opinions on the role of new and old media in science communication, debating and discussing the issues with the other panel members – Felicity Mellor and Jon Turney – and a number of contributors from the floor. Andrew Williams’ media keynote lecture preceded the “Researchers facing the media” panel session and was full of important and troublesome insights into just how science can be distorted (for good or bad) through the lens of the media.

But it was the first panel session of the conference, on the science-policy interface, that got me somewhat hot under the collar. (Well, OK, I was wearing a t-shirt so perhaps this isn’t the best metaphor…). That’s because that particular panel provided a telling insight into the gulf that still exists between natural and social scientists when it comes to the interpretation and contextual underpinnings of scientific data. Until we find a way to reconcile views spanning this gulf then we’re going to continue to exist in our silos, as two distinct cultures, arguably even more divided within the sciences than CP Snow could ever have envisaged for our separation from the arts and humanities.

The panel featured a ‘robust’ exchange of views – if you’ll excuse my borrowing of a hoary old euphemism – on the interpretation of scientific data and just how it is used to inform political debate and decisions. Chris Tyler, of the Parliamentary Office of Science and Technology, forcefully put forward his view that we can never consider scientific results in isolation from the political process. Sheila Jasanoff, Professor of Science and Technology Studies at Harvard, had earlier made very similar comments in the light of engaging presentations made by Daniele Fanelli and Beth Taylor on the interface between scientific research and policymaking. The overall tone of the debate is perhaps best summed up in this tweet from Roger Pielke (who is also speaking at the conference today in the “Challenging Established Science” panel):

Fanelli made an impassioned argument countering the idea that scientific evidence must always be considered in the context of its political framing. His comments certainly resonated with me, and I’d be rather surprised if what he said didn’t also strike a chord with the other physical/life scientists in the audience. We spend our lives aiming to do experiments in as disinterested a fashion as possible. It therefore rankles to be told that objective – and I use that word unashamedly – scientific evidence is nothing more than opinion.

For my colleagues in sociology and science and technology studies, I should stress that I am not for one second suggesting that scientists are immune to social biases. John Ziman, physicist-turned-sociologist, rightly disparaged the idea that scientists are always disinterested seekers of the truth, describing it as “the Legend”. Nor am I suggesting that data interpretation is not part and parcel of the scientific method (as Neuroskeptic argues convincingly).

The discussion yesterday, however, dangerously strayed very close at times to the ‘cultural relativism’ that was so successfully lampooned by Alan Sokal back in the nineties. Yes, scientific evidence must be considered as just one element – and, unfortunately, it’s often a very small element – of the political process. It would be naïve, at best, to argue otherwise. But the entire rationale for scientific research is underpinned by the understanding that we, as scientists, should always aim to put aside those socio-political and cultural biases. Otherwise, objective scientific evidence is reduced to pure opinion. Newton’s laws of motion, E=mc2, the Schrödinger equation, the speed of light, and the first and second laws of thermodynamics are not culturally or politically determined. Those same laws are just as valid for a race of small blue furry creatures from Alpha Centauri as they are for us.

Or, as Sokal famously put it,

“…anyone who believes that the laws of physics are mere social conventions is invited to try transgressing those conventions from the windows of my apartment. (I live on the twenty-first floor.)”

Image: The Shard in London, currently the European Union’s tallest building and a prime location to test the idea that the laws of gravity are merely an opinion. Credit: https://www.maxpixel.net/Skyscraper-Shard-Architecture-London-Landmark-949752 

The vacuity of ‘excellence’

 

800px-Dark_Rift_2012

Originally published at physicsfocus

This post has been simmering and in gestation for quite a while. This week, however, a number of documents arrived in my inbox to finally catalyse me into putting pen to paper. (Literally. I wrote this out long-hand before editing while typing it up. If you think that it’s vitriolic and ranty now, you should have seen the first scribbled draft…)

The source of my irritation? Well, take a look at the five statements below, each culled from the website of a leading UK university. (The names of the institutions have been omitted to protect the guilty).

 “Through research of international excellence, to increase significantly the range of human knowledge and understanding…”

“We seek the highest distinction in research and scholarship and are committed to excellence in all aspects of education and transmission of knowledge.”

“By bold innovation and excellence in all that we do, we make both knowledge and discoveries matter. “

“.. we want to rise further and be amongst the very few premier global universities. We will achieve this through the excellence of our research and teaching…”

“The University …. combines academic excellence with an innovative and entrepreneurial approach to research…”

Do you see a common theme here? Yep, it’s that word – “excellence”. (Those are just five examples out of countless others. Go to any university website and type in “excellence” into the search box – you’ll be swamped by links.)

It’s not only the marketing blurb for universities that is riddled with references to excellence. The tagline for Research Councils UK is “Excellence with Impact”; UK academics have just been subjected to the rigours of data collection for HEFCE’s Research Excellence Framework (and the associated game-playing over just who is “excellent” and who isn’t); OFSTED has its “excellence gateway”; the NHS is “energised for excellence”, and even the British Parking Association celebrates parking excellence.

But what does “committed to excellence” actually mean?

Here’s what it means: Nothing. Absolutely nothing. It’s nothing more than the worst form of tedious, clichéd, vacuous, buttock-clenchingly awful marketing hyperbole.

What else is a university, or any type of organisation, going to do than try to be excellent? Strive for mediocrity?  Pursue adequate performance? Try to be a little better than the rest, but not aim too high?

Ugh.

Seventeen years ago, in The University in Ruins, Bill Readings described the many problems resulting from academia’s reliance on the nebulous concept of excellence. (Thanks to my colleague at Nottingham, John Holmwood, for making me aware of Readings’ excellent book). Here’s one particularly insightful argument:

“The point is not that no-one knows what excellence is, but that everyone has his or her own idea of what it is. And once excellence has been accepted as an organizing principle, there is no need to argue about differing definitions… if a particular department’s kind of excellence fails to conform, then that department can be eliminated without apparent risk to the system.”

(In this context, changing the name of the UK’s national research assessment exercise to the Research Excellence Framework makes a great deal of sense.)

Readings goes on to discuss what he describes as the “empty notion of excellence”. There’s an important concept in semiotics which captures this vacuity: the empty (or floating) signifier. An empty signifier is literally meaningless – it doesn’t represent any particular object or meaning which is universally agreed. “Excellence” is as good an example of an empty signifier as one could hope to find.

It takes a particularly insidious form of hypocrisy for UK universities to argue that they will develop the critical thinking skills of their students while at the same time they proclaim a commitment to excellence in everything they do. Laurie Taylor’s wonderful spoof University of Poppleton, with its commitment to being “fair to middling at everything”, at least has the advantage of a clear and original mission statement.

Image: Space is mostly vacuum, but it’s not nearly as empty as meaningless commitments to “excellence”. Credit: NASA/ESA

Peer review in public: Rise of the cyber-bullies?

cyborgs.png

Originally published at physicsfocus.

A week ago in a news article in Science – and along with my colleagues and collaborators, Julian Stirling and Raphael Levy – I was accused of being a cyber-bully. This, as you might imagine, was not a particularly pleasant accusation to face. Shortly following publication of the piece in Science, one of the most popular and influential science bloggers on the web, Neuroskeptic, wrote an insightful and balanced blog post on what might be best described as the psychology underpinning the accusation. This prompted a flow of tweets from the Twitterati…

tweets

As one of the scientists at the eye of the storm, I wanted to take some time to explain in this blog post just how this unfortunate and distressing situation (for all involved) arose because it has very important implications for the future of peer review. I’ll try to do this as dispassionately and magnanimously as possible, but I fully realise that I’m hardly a disinterested party.

The science and the censure

The back-story to the claim of cyber-bullying is lengthy and lively. It spans almost 30 published papers (very many in the top tier of scientific journals – see the list here), repeated refusals to provide raw data and samples to back up those published claims, apathetic journal editors (when it comes to correcting the scientific record),  strong public criticism of the research from a PhD student initially involved in the contested work, years of traditional peer review before a critique could make it into the literature, a bevy of blog posts, a raft of tweets, and, most recently, the heaviest volume of comments on a PubPeer paper to date.

For those of you who have the stamina to follow the entire, exhausting story, Raphael Levy has recently put together a couple of very helpful compendia of blog posts and articles. I’ve given myself the challenge here at physicsfocus of condensing all of that web traffic down into a short(-ish) Q&A to provide a summary of the controversy and to address the questions that crop up repeatedly. As a case study in post-publication peer review (PPPR), there is an awful lot to learn from this controversy.

Q. What scientific results are being challenged?

In 2004, Francesco Stellacci and co-workers published a paper in Nature Materials in which they interpreted scanning tunnelling microscopy (STM) images of nanoparticles covered with two different types of molecule as showing evidence for stripes in the molecular ‘shell’. They followed this paper up with a large number of other well-cited publications which built on the claim of stripes to argue that, for example, the (bio)chemistry and charge transport properties of the particles are strongly affected by the striped morphology.

Q. How has the work been criticised?

In a nutshell, the key criticism is that imaging artefacts have been interpreted as molecular features.

In slightly more detail…

  • The stripes in the images arise from a variety of artefacts due to poor experimental protocols and inappropriate data processing/analysis.
  • The strikingly clear images of stripes seen in the early work are irreproducible (both by Stellacci’s group and their collaborators) when the STM is set up and used correctly.
  • The data are cherry-picked; there is a lack of appropriate control samples; noise has been misinterpreted, and there is a high degree of observer bias throughout.
  • Experimental uncertainties and error bars are estimated and treated incorrectly, from which erroneous conclusions are reached.

That’s still only a potted summary. For all of the gory detail, it’s best to take a look at a paper we submitted to PLOS ONE at the end of last year, and uploaded at the same time to the Condensed Matter arXiv and to PubPeer.

Q. …but that’s just your opinion. You, Levy, and Stirling could be wrong. Indeed, didn’t leading STM groups co-author papers with Francesco Stellacci last year? Don’t their results support the earlier work?

First, I am not for one second suggesting that I don’t get things wrong sometimes. Indeed, we had to retract a paper from Chem. Comm. last year when we found that the data suffered from an error in the calibration of the oscillation amplitude of a scanning probe sensor. Embarrassing and painful, yes, but it had to be done: errare humanum est sed perserverare diabolicum.

The bedrock of science is data and evidence, however, not opinion (although, as Neuroskeptic highlighted, the interpretation of data is often not cut-and-dried). It took us many months to acquire (some of) the raw data for the early striped nanoparticle work from the authors, but when it finally arrived, it incontrovertibly showed that STM data in the original work suffered from extreme feedback loop instabilities which are very well-known to produce stripes aligned with the (slow) scan direction. This is exactly what is seen in this (from the very first paper on striped nanoparticles):

stripes.png

What is remarkable is that Francesco Stellacci’s work with those leading STM groups last year not only doesn’t support the earlier data/analysis, it clearly shows that images like that above can’t be reproduced when the experiment is done correctly. (Note that I contacted those groups by e-mail more than a week in advance of writing this post. They did not respond.)

But that’s more than enough science for now. The technical aspects of the science aren’t the focus of this post (because they’ve been covered at tedious length previously).

Q. Why do you care? For that matter, why the heck should I care?

I care because the flaws in the striped nanoparticle work mislead other researchers who may not have a background in STM and scanning-probe techniques. I care because funding of clearly flawed work diverts limited resources away from more deserving science. I care because errors in the scientific record should not stand uncorrected – this severely damages confidence in science. (If researchers in the field don’t correct those errors, who will?). And I care because a PhD student in the Stellacci research group was forced into the unfortunate position of having to act as a whistleblower.

If you’re a scientist (or, indeed, a researcher in any field of study), you should care because this case highlights severe deficiencies in the traditional scientific publishing and peer review systems. If you’re not, then you should care because, as a taxpayer, you’re paying for this stuff.

Q. But can’t you see that by repeatedly describing Francesco Stellacci’s work as “clearly flawed” online, he may well have a point about cyber-bullying?

Can I understand why Francesco might feel victimised? Yes. Can I empathise with him? Yes, to an extent. As a fellow scientist, I can entirely appreciate that our work tends to be a major component of our self-identity and, as Neuroskeptic explains, a challenge to our research can feel like a direct criticism of ourselves.

But as I said in response to the Science article, to describe criticism of publicly-funded research results published in the public domain as cyber-bullying is an insult to those who have had to endure true cyber-bullying. If public criticism of publicly-funded science is going to be labelled as cyber-bullying, then where do we draw the line? Should we get rid of Q&A sessions at scientific conferences? Should we have a moratorium on press releases and press conferences in case the work is challenged? Should scientists forgo social media entirely?

Q. Don’t you, Levy, and Stirling have better things to do with your time? Aren’t you just a little, ahem, obsessive about this?

Yes, we all have other things to do with our time. Julian recently submitted his thesis, had his viva voce examination, passed with flying colours, and is off to NIST in March to take up a postdoctoral position. Raphael was recently promoted and is ‘enjoying’ the additional work-load associated with his step up the career ladder. And I certainly could find other things to do.

I can only speak for myself here. I’ve already listed above a number of the many reasons why I care about this striped nanoparticle issue. If the work was restricted to one paper in an obscure journal that no-one had read then I might be rather less exercised. And I certainly don’t make a habit of critiquing other groups’ work in such forensic detail. (Nor have I got a particular axe to grind with Francesco – I have never met the man and am certainly not pursuing this in order to “tarnish his reputation”.)

But the striped nanoparticle ‘oeuvre’ is riddled with basic errors in STM imaging and analysis – errors that I wouldn’t expect to find in an undergraduate project report, let alone in Nature Publishing Group and American Chemical Society journals. This is why we won’t shut up about it! That this research has been published time and time again when there are gaping holes in the methodology, the data, and the analyses is a shocking indictment of the traditional peer review system.

Q. But then surely the best way to deal with this is through the journals, rather than scrapping it out online?

Raphael Levy spent more than three years getting a critique of the striped nanoparticle data into print before he started to blog about it. I’ve seen the exchange of e-mails with the editors for just one of the journals to which he submitted the critique – all taken, it runs to thirty pages (over ninety e-mails) over three years. While this was going on, other papers based on the same flawed data acquisition and analysis processes were regularly being published by Francesco and co-workers. There is no question that traditional peer review and the associated editorial processes failed very badly in this case.

But is PPPR via sites such as PubPeer the way forward? I have previously written about the importance of PPPR (in this article for Times Higher Education), and some of my heroes have similarly sung the praises of online peer review. I remain of the opinion that PPPR will continue to evolve such that it will be de rigueur for the next generation of scientists. However, the protracted and needlessly tortuous discussion of our paper over at PubPeer has made me realise that there’s an awful lot of important work left to do before we can credibly embed post-publication peer review in the scientific process.

Although PubPeer is an extremely important – indeed, I’d go so far as to say essential and inevitable – contribution to the evolution of the peer review system, the approach as it stands has its flaws. Moderation of comments is key, otherwise the discussion can rapidly descend into a series of ad hominem slurs (as we’re seeing in the comments thread for our paper). But even if those ad hominems are sifted out by a moderator, those with a vested interest in supporting a flawed piece of work – or, indeed, those who may want to attack a sound paper for reasons which may not be entirely scientific – can adopt a rather more subtle approach, as Peer 7 points out in response to a vociferous proponent of Stellacci et al’s work:

“You are using a tactic[al] which is well known by online activists which consists of repeating again and again the same series of arguments. By doing so you discourage the reasonable debaters who do not have the time/energy to answer these same arguments every day. In the same time, you instil doubt in less knowledgeable people’s mind who could think that, considering the number of your claims, some might be at least partly true.”

Moderation to identify this type of ‘filibustering’ will not come cheap and it will not be easy – there will always be the issue of finding truly disinterested parties to act as moderators. A colleague (not at Nottingham, nor in the UK) who wishes to remain anonymous – the issue of online anonymity is certainly vexed – and who has been avidly following the striped nanoparticle debate at PubPeer, put it like this in an e-mail to me:

The way this thing is panning out makes me actually more convinced that a blog is not a proper format for holding scientific debates. It might work to expose factually proven fraud. The peer-reviewed, one-argument-at-a-time format does one fundamental thing for the sanity of the conversation which is that it “truncates” it. It serves the same purpose of the clock on politicians’ debates. And protects, at least to an extent the debater from Gish gallop[s]… and the simple denial techniques. Just because you cannot just say that somebody is wrong on a paper and get away with it. At least it is harder than on a blog

As I said in that Times Higher article, much of the infrastructure to enable well-moderated online commentary is in principle already in place for the traditional journal system. We need to be careful not to throw the baby out with the bathwater in our efforts to fix the peer review system: PPPR should be facilitated by the journals – in, of course, as open a fashion as possible – and embedded in their processes instead of existing in a parallel online universe. When it takes more than three years to get criticism of flawed research through traditional peer review channels, the journal system has to change.

—-

P.S. The image we wanted to use for this post was this, which, as the Whovians amongst you will realise, would have rather neatly tied in with the title. The BBC refused permission to use the image. If they’re going to be like that, they’re not getting their Tardis back

Image: Scientists online want your clothes, your boots and your motorcycle. Or maybe just to correct the scientific record. Credit: DarkGeometryStudios/Shutterstock

The spirit-crushing impact of impact

Graphene_xyz

A couple of years ago I contributed a chapter entitled “Science as a Public Good” to a book called A Manifesto for the Public University, edited by John Holmwood, a colleague in the School of Sociology and Social Policy here in Nottingham. As a prelude to the following diatribe   cri de coeur reasoned dissection of the impact agenda, here’s the opening paragraph from that chapter (you can read both the chapter and, indeed, the entire book, for free via the preceding links):

“I have a confession to make. It’s a difficult admission in the current funding climate for academics in the UK, but here it is: I am a scientist. Not an engineer. Not a technologist. And certainly not an entrepreneur. I pursue basic research into fundamental questions about the properties of matter on a variety of different length scales (ranging, in my case, from sub-atomic to sub-millimetre dimensions), in common with a very large number of my colleagues working in the physical and life sciences in British universities. Whether or not this research can be translated into a marketable product, exploited as profitable intellectual property (IP), or applied in technology is not what motivates me. My motivation, again in common with the majority of academic scientists in the UK,1 lies in improving our understanding of nature, generating (not protecting) new knowledge, and disseminating my findings to other scientists, students and society at large.”

Clare Burrage, a Royal Society University Fellow in the Particle Theory Group at Nottingham – and, as I mentioned in yesterday’s post, a fellow participant in the Royal Society MP-Scientist pairing scheme – managed to distil the essence of the paragraph above into a pithy one-sentence question following a presentation by Alexandra Saxon, Head of Research Council UK’s Strategy Unit, in Westminster on Tuesday afternoon this week.

Before I reveal Clare’s question, here’s a slide which is wholly representative of the content of Alexandra’s talk:

impact.png

The entire focus of the RCUK presentation was on engagement with business and industry. (It’s worth noting that this was given to a room-full of scientists). Fundamental science that didn’t link to some aspect of commercialisation or direct socioeconomic impact didn’t get a look-in.

Clare’s question following the presentation neatly summed up the feelings of quite a few in the room. I had to struggle to maintain my usual dignified silence (*cough*) and not cheer out loud when Clare asked this:

“I’m a researcher working on theoretical particle physics – where do I fit in?”

To be fair to Alexandra, she, along with a number of other RCUK representatives, regularly faces the music in front of crowds of academics and it is to her, and her colleagues’, immense credit that there is a strong willingness to engage with disgruntled academics. It’s a shame, however, that the willingness to interact with rank-and-file academics sometimes isn’t quite as evident further up the chain of the RCUK hierarchy. The Engineering and Physical Sciences Research Council, for example, has recently had its wrist slapped about failures to consult sufficiently with academics about its policies (but, again to its credit, has implemented policy changes to deal with this criticism).

Alexandra’s talk was certainly not the first time the spectre of impact reared its head during the days we spent in Westminster. The previous day’s panel featuring Robert Winston, Alan Malcolm, and Chris Tyler amongst others, also touched on the subject of the impact agenda.

I’ve heard the justifications as to why academics should embrace the impact agenda, including those offered by Robert Winston and Alexandra Saxon, countless times before. Let’s take a look at the five most common reasons used to incentivise scientists about impact:

1. Without the impact agenda, the science budget would have been cut dramatically – the flat-cash settlement we have ‘enjoyed’ over the last few years would not have been possible. Anti-impact sentiment threatens the science budget.

First, there is a very troublesome aspect of this argument which suggests that academics should keep their heads well below the parapet and just be happy that they have been protected from the vicious spending cuts imposed across the rest of the public sector. I’ve previously described this as a supine position to adopt, and, without wanting to open old wounds, I remain of that opinion. If academics see problems with just how the research councils distribute funding then it is highly questionable to suggest that they should put those concerns to one side so as not to affect the funding flow. Whatever happened to the traditional university role of speaking truth to power?

The broader point, however, is that expecting academics, regardless of their discipline, to describe the socioeconomic impact of their work in advance of the research project being carried out is antithetical to the exploratory nature of fundamental research. As the University of Nottingham put it in a response to a consultation on the introduction of economic impact criteria into the peer review process back in 2007:

“[This] appears to fly in the face of the purpose of “research” within universities … could stifle highly imaginative, original and creative work, or lead to dubious, often irrefutable, claims in many areas of science – especially fundamental/“blue skies” research.

(Sorry to be so parochial – you can find very similar statements from Cambridge and Glasgow, for example, in the chapter from A Manifesto for the Public University linked to above).

Moreover, why is asking academics to fill in a Pathways to Impact statement (and, for EPSRC, an account of National Importance) at the proposal stage, i.e. prior to starting a research project, the best way to demonstrate the socioeconomic impact of research to government?

2. You’re a publicly funded academic. You can’t just stay in your ivory tower – you have to think about the societal impact of your work.

If there’s one justification for the impact agenda that really gets my goat and grinds my gears, it’s this one. The impact agenda, for the reasons detailed in that chapter from AMPU, was not introduced because the research councils/HEFCE/government suddenly became extremely concerned that not enough academics were engaging with the public. It was introduced as a direct response to a series of government reports on the need to change the culture in university departments so as to make academia more responsive to business and industry needs.

I, along with a significant majority of my colleagues, am intensely aware of my obligations as a publicly funded academic, and spend a significant amount of time on public engagement and outreach. For example, when I finish writing this I’m boarding a train to Loughborough to give a talk to a school there on the relationship between rock music and quantum physics. (I’ll also not pass up this opportunity to plug Brady Haran’s channels, and Sixty Symbols in particular).

One can also very easily make the counter-argument that, by incentivising academics to interact closely with business and industry, the disinterested and independent qualities of academic research are being progressively eroded. That is, the impact agenda, like many aspects of the coalition government’s policies is not about improving the public good character of university science at all: it’s about making the public sector responsive to private business.

I am, of course, not for one minute suggesting that all academic-industry collaborations are compromised by the need to improve the company’s bottom line. Nonetheless, there are some shocking examples of where academic research has been distorted badly due to commercial pressures. I referred yesterday to the tagline of George Monbiot’s recent article: “Government policy in Britain, Canada and Australia is crushing academic integrity on behalf of corporate power”. The impact agenda certainly doesn’t help to bolster the integrity of academic research in the face of commercial pressures.

3. The government expects a return on its investment in university research.

Richard Jones, PVC for Research and Innovation at the University of Sheffield, has pointed out that the key reason why there is intense pressure on UK academics to demonstrate the impact of their work is that private sector investment in R&D in the UK (as a percentage of GDP) heavily lags behind that of other OECD nations. Thus, academia is expected to pick up the slack for this lack of investment from the private sector.

The key thing here is that fundamental scientific research is but one component of a highly complex innovation ecosystem. Although the motivation for academics to carry out basic, so-called curiosity-driven science is very often not the economic impact of the work – we instead are focussed on understanding fundamental aspects of nature – it is clear that the most disruptive innovations are exceptionally unlikely to stem from work which is focused on near-market impact. In this sense, by progressively skewing the research base towards commercial, near-market R&D, the total return on government investment may well be damaged, not enhanced.

Expecting academics to handle all aspects of the innovation system – from fundamental research to manufacturing and “product to market” – is in any case an exceptionally naïve strategy. Andre Geim pointed out in the Guardian earlier this week that perhaps the government should look somewhere other than the universities when apportioning blame for the so-called “valley of death” between fundamental research and commercial exploration. Jones has made this argument convincingly for quite some time.

Moreover, an exceptionally important – arguably, the most important – contribution to the return on investment for government spending on university science is the “human capital” we produce. David Willetts clearly recognised this when he spoke of the “absorptive capacity” of our society and innovation systems shortly after the Coalition government came to power.

4. “But it’s not all about economic impact – public engagement is impact as well.”

It is laudable that the research councils recognise that public engagement and outreach are essential components of the impact agenda. Why then do RCUK representatives focus so heavily on links with business and industry? In the talk on Tuesday – and this is true of very many other talks from research council representatives I’ve seen – lip service, at best, is paid to the remarkably inspiring fundamental science stemming from UK academia. Geim and Novoselov’s Nobel prize-winning discovery of graphene – and the government’s investment in commercialisation of the carbon wunderkind – are regularly highlighted without mention that graphene stemmed not from the impact agenda or directed programmes of the type to which EPSRC is now fully committed, but from exploratory, curiosity-driven fundamental science with no commercial application in mind.

One could also ask why, if EPSRC, for example, is so committed to public engagement, it no longer has a dedicated budget to fund public engagement projects.

5. “We’re just as committed to fundamental science as we ever were”

Lilian Greenwood, the MP with whom Clare Burrage and myself are paired in the Royal Society scheme, asked an excellent question of David Willetts yesterday morning during the Business, Innovation, and Skills questions session:

“Does the Minister agree that investing in fundamental research is vital to inspire the next generation of scientists and engineers and to create conditions for the serendipitous discoveries of the future?”

David Willetts responded:

I completely agree with the Honourable Lady, which is why the Government support fundamental research. Only last week I went to the launch of £250 million of public money for centres of doctoral training run by the Engineering and Physical Sciences Research Council”

Similarly, research council and HEFCE representatives repeatedly claim that there has been no change in their support of fundamental research.

The key question here is just what is meant by fundamental research. It is interesting that each and every one of the new Centres for Doctoral Training to which Willetts refers has industrial partners (as compared to 50% of CDTs involving industrial collaboration in the previous round). Moreover, there was a very strong steer from EPSRC that those CDTs should have training programmes “co-created” with industry. One might reasonably ask just what Mr. Willetts, BIS, RCUK, and HEFCE understand by the term “fundamental research”.

As I’ve also harped on about previously, it is rather difficult to understand RCUK’s commitment to fundamental research when the #1 tip in its Top Ten list of tips on how to complete a grant application is “Draft the impact summary very early in your preparation, so that it informs the design of your research”. That’s a great piece of advice if you’re writing a proposal to do applied, near-market research, or R&D for a company. It’s certainly not how fundamental science proceeds. Indeed, one could credibly argue that it’s a distortion of the scientific method.

This has been a long (but cathartic!) post, for which my apologies. I would like to close by stating that I’m aware that a lot of what I’ve written above could prompt strong responses. I would welcome the opportunity to debate these issues in the comments section below. Given RCUK’s and HEFCE’s commitment to public accountability in all areas of research, comments from research/funding council representatives would be particularly welcome.

Image: The molecular structure of graphene. The “wonder material” was discovered from curiosity-driven fundamental science without a commercial application in mind. Credit: Dr Thomas Szkopek