Not everything that counts can be counted

8539517000_9a44748db4_n

First published at physicsfocus.

My first post for physicsfocus described a number of frustrating deficiencies in the peer review system, focusing in particular on how we can ensure, via post-publication peer review, that science does not lose its ability to self-correct. I continue to rant about discuss and dissect the issue of post-publication peer review in an article in this week’s Times Higher Education, “Spuriouser and Spuriouser”. Here, however, I want to address some of the comments left under that first physicsfocus post by a Senior Editor at Nature Materials, Pep Pamies (Curious Scientist in the comments thread). I was really pleased that a journal editor contributed to the debate but, as you might be less than surprised to hear, I disagree fundamentally with Pep’s argument that impact factors are a useful metric. As I see it, they’re not even a necessary evil.

I’m certainly not alone in thinking this. In an eloquent cri de coeur posted at his blog, Reciprocal Space, last summer, Stephen Curry bluntly stated, “I am sick of impact factors. And so is science”. I won’t rehearse Stephen’s arguments – I strongly recommend that you visit his blog and read the post for yourself, along with the close to two-hundred comments that it attracted – but it’s clear from the Twitter and blog storm his post generated that he had tapped into a deep well of frustration among academics. (Peter Coles’ related post, The Impact X-Factor, is also very well worth a read.)

I agree with Stephen on almost everything in his post. I think that many scientists will chuckle knowingly at the description of the application of impact factors as “statistically illiterate” and I particularly liked the idea of starting a ‘smear campaign’ to discredit the entire concept. But he argues that the way forward is:

“…to find ways to attach to each piece of work the value that the scientific community places on it though use and citation. The rate of accrual of citations remains rather sluggish, even in today’s wired world, so attempts are being made to capture the internet buzz that greets each new publication; there are interesting innovations in this regard from the likes of PLOS, Mendeley and altmetrics.org.”

As is clear from the THE article, embedding Web 2.0/Web 3.0/Web n.0 feedback and debate in the peer review process is something I fully endorse and, indeed, I think that we should grasp the nettle and attempt to formalise the links between online commentary and the primary scientific literature as soon as possible. But are citations – be they through the primary literature or via an internet ‘buzz’ – really a proxy for scientific quality and the overall value of the work?

I think that we do science a great disservice if we argue that the value of a paper depends only on how often other scientists refer to it, or cite it in their work. Let me offer an example from my own field of research, condensed matter physics – aka nanoscience when I’m applying for funding – to highlight the problem.

Banging a quantum drum

Perhaps my favourite paper of the last decade or so is “Quantum Phase Extraction in Isospectral Electronic Nanostructures” by Hari Manoharan and his co-workers at Stanford. The less than punchy title doesn’t quite capture the elegance, beauty, and sheer brilliance of the work. Manoharan’s group exploited the answer to a question posed by the mathematician Mark Kac close to fifty years ago: Can one hear the shape of a drum? Or, if we ask the question in rather more concrete mathematical physics terms, “Does the spectrum of eigenfrequencies of a resonator uniquely determine its geometry?”

For a one dimensional system the equivalent question is not too difficult and can readily be answered by guitarists and A-level physics students: yes, one can ‘hear’ the shape, i.e. the length, of a vibrating string. But for a two dimensional system like a drum head, the answer is far from obvious. It took until 1992 before Kac’s question was finally answered by Carolyn Gordon, David Webb, and Scott Wolpert. They discovered that it was possible to have 2D isospectral domains, i.e. 2D shapes (or “drum heads”) with the same “sound”. So, no, it’s not possible to hear the shape of a drum.

What’s this got to do with nanoscience? Well, the first elegant aspect of the paper by the Stanford group is that they constructed two-dimensional isospectral domains out of carbon monoxide molecules on a copper surface (using the tip of a scanning tunnelling microscope). In other words, they built differently shaped nanoscopic ‘drum heads’, one molecule at a time. They then “listened” to the eigenspectra of these quantum drums  by measuring the resonances of the electrons confined within the molecular drum head and transposing the spectrum to audible frequencies.

So far, so impressive

But it gets better. A lot better.

The Stanford team then went on to exploit the isospectral characteristics of the differently shaped quantum drum heads to extract the quantum mechanical phase of the electronic wavefunction confined within. I could wax lyrical about this particular aspect of the work for quite some time – remember that the phase of a wavefunction is not an observable in quantum mechanics! – but I encourage you to read the paper itself. (It’s available via this link, but you, or your institution will need a subscription to Science.)

I’ll say it again – this is elegant, beautiful, and brilliant work. For me, at least, it has a visceral quality, just like a piece of great music, literature, or art; it’s inspiring and affecting.

…and it’s picked up a grand total of 29 citations since its publication in 2008.

In the same year, and along with colleagues in Nottingham and Loughborough, I co-authored a paper published in Physical Review Letters on pattern formation in nanoparticle assemblies. To date, that paper has accrued 47 citations. While I am very proud of the work, I am confident that my co-authors would agree with me when I say that it doesn’t begin to compare to the quality of the quantum drum research. Our paper lacks the elegance and scientific “wow” factor of the Stanford team’s publication; it lacks the intellectual excitement of coupling a fundamental problem (and solution) in pure mathematics with state-of-the-art nanoscience; and it lacks the sophistication of the combined experimental and theoretical methodology.

But yet our paper has accrued more citations.

You might argue that I have cherry-picked a particular example to make my case. I really wish that were so but I can point to many, many other exciting scientific papers in a variety of journals which have attracted a relative dearth of citations.

Einstein is credited, probably apocryphally, with the statement “Not everything that counts can be counted, and not everything that can be counted counts”. Just as multi-platinum album sales and Number 1 hits are not a reliable indicator of artistic value (note that One Direction has apparently now out sold The Beatles), citations and associated bibliometrics are not a robust measure of scientific quality.

Image credit: https://www.flickr.com/photos/bigleaftropicals/8539517000 

Author: Philip Moriarty

Physicist. Rush fan. Father of three. (Not Rush fans. Yet.) Rants not restricted to the key of E minor...

26 thoughts on “Not everything that counts can be counted”

  1. Nice post once again.

    We recently blogged about our data analysis of the impact factor here. The general conclusion is the same as the one you reach above. Impact factor is not a measure of quality — it is more a measure of context.

    But as flawed as it is, impact factor and citation counts are still probably the best measure we have to evaluate research. They are still preferable to counting how many times a paper has been tweeted, for instance.

    As we see it, the way forward is to generate more direct metrics during the peer review process; to get the initial peer reviewers and other readers of the paper to directly evaluate the quality and significance of the paper.

    We thus see a modern peer review system as having two key aspects:
    – ratings of specific dimensions of the paper (eg. quality and significance) to allow for immediate and accurate analysis
    – a written component to unlock (and summarize) the nuances of a complex work.

    This is what we’re experimenting with at the moment at Publons. See our survey for a sneak preview of the peer review format we’ll be rolling out shortly.

    Like

  2. Impact factors are IMHO not just imperfect, they are nonsense.

    A story. I have just completed the most accurate temperature measurement in all of human history. The result will form the basis of the redefinition of the kelvin in a year or two’s time. After the redefinition, every single temperature measurement in the world will be (at least in part) scaled to our measurement. We thought this was important for people to know about and stepped outside of our metrological ghetto and submitted it to Nature and Nature Physics. They were not interested.

    At first I was puzzled, but then I realized that the impact factor works two ways. People want to publish in ‘prestigious’ journals because of the high impact factor. However the journals can only accept papers that will be multiply referenced. They are thus compelled to publish articles in fashionable fields. When I realised the nature of the game (no pun intended) I didn’t feel so bad.

    And I invite you take a look through the journals and ask yourself honestly if this is really la creme de la creme of academic research. I don’t think it is. I think there is an occasional important paper, but the rest is just fashionable research. I understand the search for an ‘objective’ measure of quality, but as anyone who has ever read Zen and the art of Motorcycle Maintenance will tell you, its impossible. The use of any metric other than the judgement of a human being expert in the field is open to gaming – and many academic see their activities as a game to win rather than as a calling to participate.

    Anyway. My 2p there.

    M

    Like

  3. I am not surprised that the paper on fingering patterns in dewetting at the nanoscale has gathered more citations than that on quantum isospectral nanostructures. Simply put, there is more scientific output (and more people working) on patterns at the nanoscale than on phase information of wave functions.

    Citations are likely a good measure for quality when comparing papers within a sufficiently small field. In larger contexts, citations are more a measure of impact. As with music, the number of times a song has been bought or broadcasted is a good indicator of the impact of the song, not necessarily of its quality. There is of course high-quality music that is only listened by niche audiences.

    You wrote: “I think that we do science a great disservice if we argue that the value of a paper depends only on how often other scientists refer to it, or cite it in their work.”

    As with the value of a song or a piece of art, the value of a piece of scientific work is a subjective measure, and metrics (be it citations, highest h-index among the authors, journal where it is published or number of downloads, for instance) give us a variety of indicators of such value, be it impact, quality, significance, implications or usefulness.

    Similarly, movie ratings in the IMDB, the price of a portrait in an auction, or the number of stars a book accrues in Amazon, an app in an app store, or a restaurant in Yelp, these are all metrics that we rely on, and they are useful most of the time. Yet the best metric is of course seeing the movie, reading the book, eating in the restaurant or admiring the portrait. And a recommendation from a trusted friend may be second best and trump all those metrics. But, in the absence of trusted recommendations and time to watch, read or eat out as much as one would like, those stars and points are generally helpful in deciding how to prioritize our time.

    As I argued here,

    1. We are a competitive bunch fighting for limited resources.
    2. There is too much to read and too little time to sift through what may be relevant.
    3. There is no available easy-to-grasp, accurate metric for the quality and/or relevance of a paper.

    Given premisses 1 and 2, the need for metrics and rankings becomes obvious.

    And as I wrote in the Nature Materials editorial, impact factors are useful for journals, not for individual papers or scientists. Indeed, this plot makes it obvious that impact factors predict 5-year median citations really well.

    Like

    1. Simply put, there is more scientific output (and more people working) on patterns at the nanoscale than on phase information of wave functions.

      First, I’d query whether that statement is actually true. Given the intense interest in quantum computing, for example, I don’t see that your claim is a ‘given’.

      Even if your statement is true, however, then this just highlights a key difficulty with using citations not only as a measure of quality but of impact/utility – how do you delineate one sub-field from another, and normalise citation counts accordingly? Answer: You can’t, in any robust fashion.

      You contradict yourself in the following paragraph:

      Citations are likely a good measure for quality when comparing papers within a sufficiently small field. In larger contexts, citations are more a measure of impact. As with music, the number of times a song has been bought or broadcasted is a good indicator of the impact of the song, not necessarily of its quality. There is of course high-quality music that is only listened by niche audiences.

      In any case, please show me the evidence for your claim that “Citations are likely a good measure for quality …within a sufficiently small field”. I can show you plenty of counter-evidence but I’ll ‘call’ you on your cards first…

      Like

      1. * A simple search on Google Scholar brings a 125 times more cites for patterns at the nanoscale than for phase information of wave functions.

        * Yes, comparing metrics between disparate fields is always difficult. Successful attempts to design normalized metrics have been done. See here for instance.

        * There is no contradiction in my paragraph.

        * There have been studies showing correlations between citations and academic success (a proxy for quality). See here for example.

        (Needless to say, the main arguments in your post are flawed.)

        Like

        1. A simple search on Google Scholar brings a 125 times more cites for patterns at the nanoscale than for phase information of wave functions.

          Thank you for that great example of the dangers of overly simplistic bibliometric searching. How about including these terms (amongst many, many others) for “patterns at the nanoscale”: “self-assembly”, “self-organisation”, “non-equilibrium pattern formation”, “nanoparticle superlattices”, “polymer dewetting”, ”

          And for “phase information…” we could include (again, among so many others) “quantum computing” (a field in which, in case you didn’t realise, the phase of the wavefunction is somewhat important), “fundamental quantum mechanics”, “phase decoherence”, “scattering”…

          Your comment only strengthens my point re. delineation of fields, which is often done on the basis of simplistic (at best) keyword analyses.

          There is indeed a contradiction in your paragraph:

          Citations are likely a good measure for quality when comparing papers within a sufficiently small field…As with music, the number of times a song has been bought or broadcasted is a good indicator of the impact of the song, not necessarily of its quality.

          In the first sentence you argue that citations are likely a good measure for quality. In the second sentence you draw an analogy between music downloads and citations – “As with music…” – and state that the number of times a song has been bought or broadcasted (i.e. ‘cited’, to use your analogy) is a good indicator of impact, not quality.

          There have been studies showing correlations between citations and academic success (a proxy for quality).

          Hmmm. I think it’s very worrying that you would claim academic success is a proxy for quality. It’s almost like using the argument from authority fallacy. I can certainly point you to a rather high profile case in nanoscience – which we’ve discussed at length previously – where very high levels of academic success have been built on very poor quality data. There are also cases exposed week in, week out at Retraction Watch where high-flying academics have got where they are via data fabrication and fraud…

          Like

          1. * The point was to compare the output of the immediate fields in which both papers belong to as a potential proxy for where the citations may come from. Comparing the big fields of self-assembly and quantum computing defeats the purpose of the exercise. Most people working in self-assembly will not be interested in your paper.

            * And no, there is absolutely no contradiction in my paragraph: “Citations are likely a good measure for quality when comparing papers within a sufficiently small field“. In larger contexts, citations are more a measure of impact. As with music, […] not necessarily of its quality”. The first sentence refers to small fields, the second to a whole research area.

            * Academic success is a proxy for quality in general. This is obvious. Otherwise, scientific committees are doing the wrong job. Of course, no one would argue against the existence of exceptions.

            This post of yours reaches a false conclusion based on anecdotal evidence. You again and again argue about specific cases in your responses to me. One can reach any desired conclusion by cherry-picking specific cases. You of course know better than this.

            Like

            1. 1.

              Comparing the big fields of self-assembly and quantum computing defeats the purpose of the exercise… Most people working in self-assembly will not be interested in your paper.

              No, of course it doesn’t defeat the purpose of the exercise. Many in the nanoparticle/nanoscience community use the terms “self-assembly” and “self-organisation” interchangeably (despite many others recognising that there’s a strong distinction between the terms). I can even point you to groups who have cited our paper who describe their work as “research into self-assembly of nanoparticles” where they mean self-assembly in the context of assembly of nanoparticle superlattices.

              My point is that by cherry-picking preferred keywords to describe the work you fail to capture research from others who describe their work in different terms. Here’s another example: someone working on zero dimensional nanostructures is also working on quantum dots and is also working on nanoparticles (and/or nanoclusters and nanocrystals). Indeed, I would argue that the vast majority of people working in the general area of condensed matter physics in 2013 are doing nanoscience of some sort. But if they prefer not to include “nanoscience” as a keyword to describe their work does this mean that they’re somehow not working in that area?

              2. Perhaps the reason I keep seeing a contradiction in that paragraph is that the idea that citations are a measure of impact when the field is large but then they somehow transmute into a measure of quality when the field is small makes absolutely no sense to me, for the reasons outlined in #1 above. How do you decide where the boundaries of fields lie? Do it on the basis of simple-minded keyword analysis and you’re on a hiding to nothing. For example, am I a surface scientist, a nanoscientist, a physical chemist, a chemical physicist, a condensed matter physicist, a solid state physicist, etc… etc…?

              3.

              This post of yours reaches a false conclusion based on anecdotal evidence.

              OK, leave aside the Stellacci case as it’s obviously quite difficult for you to comment on that in any type of disinterested fashion (given that you’re a senior editor at the journal which has published a number of Stellacci et al.’s papers). Your argument is that academic success is a proxy for quality. My argument is that there are many cases (I don’t need to cherry pick – visit Retraction Watch) where very poor quality – indeed, fraudulent – research has contributed to very high academic status.

              And while we’re on the topic of statistical analyses vs anecdotal data, it’s rather remarkable, don’t you think, that there’s a rather strong correlation between journal impact factor and retraction rate. Why do you think this might be?

              Finally, in a comment on my previous post you stated that Peer review is, obviously, not perfect, and there are correction mechanisms in place when needed. You may be interested in Raphael Levy’s recent reality check on NPG’s correction mechanisms. Not only did the editorial staff at Nature Materials not follow their own published ethical guidelines, they didn’t do me the simple common courtesy of responding to my e-mail message….

              Like

          2. * Yes, self-assembly and self-organisation are almost always interchangeable. This is obvious.

            * Once more, my argument was that the immediate subfields corresponding to the papers you cited have audiences of significantly different size. Looking for “patterns” and “nanoscale” in one case and for “phase information” and “wave functions” in the other is a reasonable option for the immediate subfields, and the difference in output from Google Scholar is 125 fold. Of course, this is just an indication in support of my argument, and a reasonable one. As I said, most of the people working in self-assembly will not be interested in your paper and thus will not cite it. Similarly, most people working in quantum computing will not be interested in the phase-information-of-wave-functions paper. Thus, comparing the size of these big fields defeats the purpose of the exercise.

            * Again, there is no contradiction in what I wrote. My whole point in Citations are likely a good measure for quality when comparing papers within a sufficiently small field. In larger contexts, citations are more a measure of impact. is that if you compare citations of papers that do not belong to the same subfield, then the size of the field often has more weight than the quality of the paper. That is, if you compare the citations to papers in the field of graphene with those in the field of polymer thermodynamics, you are mostly comparing impact (the field of graphene has a much bigger scientific output and readership). Instead, if you compare citations for papers within the same (sufficiently small) subfield, then differences in impact tend to vanish and you are more likely to compare quality.

            * I did not refer to Stellacci or to his work. I was making reference to the comparison you made between your paper and that on electron wave functions. You base your conclusions on this anecdotal evidence. Cheery-pick another example and you can get to different conclusions.

            * The correlation between IF and retraction rate may have multiple causes. In the comments to that post you link to, Richard van Noorden gives a plausible reason. Another one is that the higher the IF of the journal, the higher the impact of the papers it publishes, the higher the readership, the higher the scrutiny of the papers. If people would care as much about papers published in low IF journals, these would have to retract a higher proportion of papers. There are lots of papers published in specialists journals that may be wrong and few care if they are.

            Like

        2. By the way, Pep, it’s worth following the discussion under Peter Coles’ excellent recent post on the working methods associated with the UK’s Research Excellence Framework. (Sidenote: In the near future I am going to write a short post an extended rant for physicsfocus entitled “The Vacuity of Excellence”).

          This comment, from Dave Carter, is particularly relevant to our exchange:

          Citations may be an indicator of Significance, or they may not be, but Rigour and Originality are key, and it may well be that your most highly cited papers are not the most rigorous or original.

          Like

  4. Thanks for your comment, Daniel. I’d read your blog post previously – it’s interesting to note that Pep Pamies crops up in the comments yet again, arguing for the validity of those bibliometric measures which, entirely coincidentally, place NPG journals at the top of the publication hierarchy.

    It’s also very interesting worrying that Pep claims that there’s a direct correlation between quality and citations/impact factor. The single example I discuss in my post above is enough to lay this particular fallacy to rest.

    But I am not at all convinced that what we need is another raft of metrics. The entire point of my post is that we need to stop attempting to quantify the unquantifiable. Once a new metric is introduced, Goodhart’s law will quickly render it invalid.

    Like

  5. I think one of the other problems with using the number of citations as a proxy for the impact of the work is that not all citations are created equal. Some citations will be foundationally important to the paper and others will be asides that could probably be omitted without hurting the central argument of the paper.

    Like

  6. @Pep.

    Cheery-pick another example and you can get to different conclusions.

    But this is my entire point!! Cherry-pick the keywords you want and you can drive the bibliometrics whichever direction you like. It’s exceptionally easy to ‘game’ the statistics this way to give you the result you want. I genuinely find it extremely unsettling that you, as an editor of a leading scientific journal, have such a strong commitment to the ‘value’ of bibliometrics when it is clearly immensely problematic to, amongst many other issues, delineate between sub-fields.

    I’ll ask you again, as you avoided the question. I do work on single atom/molecule imaging, manipulation, and spectroscopy at surfaces using scanning probes. Am I a surface scientist, a nanoscientist, a physical chemist, a chemical physicist, a condensed matter physicst, a solid state physicist…etc…etc…? Which key words does my work fall under? I guarantee that if you give me five keywords I can give you another five entirely different keywords which are just as valid.

    As Colquhoun has said :

    All this shows what is obvious to everyone but bone-headed bean counters. The only way to assess the merit of a paper is to ask a selection of experts in the field.

    Nothing else works.

    Nothing.

    By the way, I’m still waiting for Nature Materials to respond to my e-mail of three weeks ago re. the NPG ethical/duties of authors guidelines . Perhaps you could have a word with Vincent Dusastre for me? I’d appreciate that.

    Thanks.

    Philip

    Like

    1. * You are attacking a straw man. How many times do I need to say that quality makes sense within a sufficiently small subfield? Nanoscience, surface science, condensed-matter physics, solid-state physics, and all the other keywords you mention refer to BIG fields. Let’s try again: my comparison was an attempt to select reasonable subfields for the two papers that you compared in your post. But, alright, let’s do a perhaps better comparison by selecting additional keywords. Comparing the Google Scholar output for ‘nanoscale AND (patterns OR patterning OR fingering OR dewetting)’ with ‘(Interferometry OR “phase information”) AND (quantum OR “wave function” OR isospectral OR isospectrality OR “phase extraction”)’ the difference is 3-fold in favour of the patterning paper.

      * Yes, assessing the merit of a paper by asking a selection of experts is a proper (if not the best) way. Yet Colquhoun, you and many others fall for the fallacy that, in practice, this is possible for ALL papers. Actually, there is too much to read, and therefore we need (imperfect) metrics that help us prioritize what to read. By the way, the number of experts that one would need to evaluate the merit of any paper in a reproducible way would almost certainly need to be higher than the number of fingers in one hand (believe me, I have statistics on how often expert reviewers agree on the merits of a paper).

      * I repeat, self-assembly and self-organisation are almost always used interchangeably. If you don’t agree with this, you are in the minority. Still, I find your patronizing verbosity on this both funny and uncanny.

      * Regarding your query on the NPG author guidelines, I am only commenting on this blog in personal capacity, so I won’t discuss here anything related to NPG from the viewpoint of an editor. Yet my personal understanding is that Nature Materialsis following the established company procedures, which may necessarily take more time that you would wish.

      Like

      1. OK, Pep, this is getting tiresome now so let’s agree to disagree. Feel free to respond to the following if you like – and if you want to have the last word – but I’m not going to pursue this any more. It’s a waste of our time and I’m sure we both have better things to do.

        But I am not attacking a straw man. The idea that one can have a rigorous definition of a sub-field on the basis of keyword analysis is simply fundamentally flawed. I can take the five keywords you used – ‘nanoscale AND (patterns OR patterning OR fingering OR dewetting)’ – and replace them with ‘nanoscale AND (branching OR instability OR colloidal OR fractal)’. And I could give you many more similar examples if I hadn’t lost the will to live discussing this with you.

        Note that EPSRC has attempted a similarly flawed delineation of the research it funds into 111 separate areas. Result? An inconsistent, and from some perspectives rather amusing, mess. Ask some EPSRC-funded researchers about the infamous ‘Bourne graph’ when you get a chance…

        I’m pleased that you found my clarification of the difference between self-assembly and self-organisation funny. But “uncanny”?! You’ve lost me.

        Yet my personal understanding is that Nature Materialsis following the established company procedures, which may necessarily take more time that you would wish.

        Sorry, but that’s a cop-out of very large proportions indeed. How long does it take to do me the courtesy of sending an e-mail to say that the matter is being considered and that the editor will be in contact in due course? Answer: No time at all. It’s simply rude not to respond.

        Philip

        You say “How many times must I say

        Like

        1. Philip, of course that ‘The idea that one can have a rigorous definition of a sub-field on the basis of keyword analysis is simply fundamentally flawed.’ I never claimed the contrary. All I said was that when one chooses a sufficiently small field (using reasonable keywords), a comparison of citations between papers are likely to be a measure of quality rather than impact. And the keywords I mentioned are reasonable (yet not rigourous!). Indeed, I tried many combinations of reasonable keywords, and in all cases the Google Scholar output was multiple times bigger for your paper than for the quantum information article.

          This is why there is no surprise in that your paper has gathered more citations, regardless of quality.

          Like

  7. One other thing, Pep. You’ll find that there’s a very large community of researchers who would strongly argue that the terms self-assembly and self-organisation are certainly not interchangeable. It surprises me, given your professional position and your scientific background, that you would argue that the interchangeability of these terms is “obvious” and that you are apparently not aware of the distinction . (See also Richard Jones’ usual eloquent take on the subject.)

    Self-assembly is a term which many reserve for processes occurring close to equilibrium and where the spatial correlation lengths are determined by the size of the molecules/nanoparticles. Self-organisation is instead a process occurring far from equilibrium where the correlation lengths in the system can far exceed the size of the components due to significant matter/energy flow. (Marangoni convection-mediated formation of nanoparticle networks is a key example).

    It’s rather a topical subject for me, although it’s some years since I last worked in the area. Philip Ball is visiting Nottingham this afternoon to give us a talk on self-organisation and pattern formation.

    Like

  8. In an earlier post ‘Are flaws in peer review someone else’s problem?’ Philip Moriarty had observed “In principle, science is indeed self-correcting. But if there are flaws in published work who fixes them?” He had also noted that “Most of researchers are apparently of the opinion that flaws in the literature are someone else’s problem (SEP).” I agree with his concluding comment, “Wrong. This is not someone else’s problem. This is our problem and we need to address it.”

    Now on April 25, 2013 at 8:47 am Curious scientist says: “There are lots of papers published in specialists journals that may be wrong and few care if they are.” Again SEP?

    Instead of squabbling over citations and impact factors, shouldn’t we be focusing on the problem of “published papers that are wrong”? Shouldn’t there be some mechanism in place under which the published papers could be openly reviewed for the benefit of readers?

    Like

    1. Instead of squabbling over citations and impact factors, shouldn’t we be focusing on the problem of “published papers that are wrong”?

      The problem is that those aren’t disconnected problems – they’re fundamentally coupled. Papers in high profile journals attract citations. This lends more credibility to the paper, particularly as a number of studies have shown that a significant percentage of authors will cite work they haven’t even read. When that paper is challenged, one of the arguments to support the ‘validity’ of the work is that it must be right because so many other researchers in the field have cited it.

      I advise you to visit Raphael Levy’s blog to get an insight into just how difficult it is to criticise papers published in high impact factor journals (whose prestige is, of course, built on citations).

      Best wishes,

      Philip

      Like

  9. I think you’re being too hard on Pep, Philip. It’s an imperfect world, citations aren’t the be-all-and-end-all, journals don’t always get it right. But there’s bigger problems in physics, and one of them is “experts in the field”. I really am not kidding about this. One day you will come to appreciate the enormity of it, and you will be appalled.

    Like

    1. Hi, John.

      But Pep’s point is that citations are the be-all-and-end-all because we don’t have enough time to actually sit down and read the papers!

      In one sense I agree with you re. experts in the field. One of my favourite Feynman quotes was his one-sentence definition of science (as given to a conference of schoolteachers when asked to define science in a sentence!): “Science is the belief in the ignorance of experts”.

      !

      All the best,

      Philip

      Like

      1. That’s not quite what he said Phil.

        Talking of Feynman and experts, take a look at this: http://youtu.be/gooez_AcZlQ

        It isn’t wrong. It’s right. Spooky action-at-a-distance is wrong. Einstein was right. I’m afraid there are issues in physics that dwarf those you’ve raised. Would you like another example? See the QED given explanation of how gamma-gamma pair production works: a photon spontaneously morphs into a fermion pair like worms from mud, and the other photon interacts with one of the fermions. So pair production occurs because pair production occurs. It’s cargo-cult nonsense masquerading as QED. Hear that turbine hum? That’s Feynman, turning in his grave.

        Like

    1. Hi, Sabine.

      Thank you for bringing my attention to those important blog posts. I hope that Pep Pamies (“Curious Scientist” in the comments thread above), Senior Editor at Nature Materials, takes the time to read your posts. He might like to ponder just why your statements below will resonate very strongly with quite a few scientists (myself included):

      In other words, the proximate cause of trying to publish in a high impact journal erodes the ultimate cause of doing good science… I wish people would pay more attention to the backreaction that comes from doing the [bibliometric] measurements and providing people with a time-saving simplified substitute for the ultimate goal of doing good science.

      Like

    2. It is indeed unfortunate that the meaning of the impact factor is too commonly extrapolated to individual papers or scientists. The IF should be used to compare journals, and it is in fact a good predictor of the median citations to original research papers. See my Nature Materials editorial for details.

      I would take the conclusions of studies that claim that the IF and citations poorly correlate with a grain of salt. Indeed, it has happened before that people have not compared apples to apples. See for instance this paper, where the effect of the near-zero citations to news articles was included in the data without explicit mention in the paper. For more details, see the discussion thread about these dodgy statistics here.

      Like

Comments are closed.