How much energy is associated with information? With knowledge? With thought?
Could we extract power solely from information, from a state of knowing?
In other words, is there a thermodynamics of information? (Don’t click on that link just yet if you want to avoid spoilers..)
Each time I receive the e-mail from Brady telling me he’s uploaded a video to which I’ve contributed, I get that familiar feeling in the pit of my stomach stemming from the worry that I might have screwed up the explanation of the physics and potentially misled those who are watching.
In a short(ish) video, we necessarily have to gloss over and/or omit lots of detail and, as a physics lecturer, this is always uncomfortable. (Deeply, deeply uncomfortable). Indeed, and as I described in a Physics World article a couple of years ago, for some time I decided to withdraw from making Sixty Symbols videos for precisely this reason. But as also discussed in that article, the ability to connect with an audience who are enthralled by, and enthusiastic about, physics, the intellectual challenge of explaining difficult concepts, and the sheer fun of working with Brady (our occasional spats notwithstanding) meant that I quickly saw the error of my ways. (As, of course, Mr Haran had predicted.)
Nonetheless, if there’s one topic that I find exceptionally difficult to put across in the short, snappy, “bite-size” YouTube format, it’s entropy. I had sworn off trying to explain the intricacies off this particular thorny concept in the Sixty Symbols style, but I keep getting drawn back to it, almost against my will — because it’s so damn interesting. When it came to thinking about thoughts, entropy and energy, we had to bite the bullet because entropy is at the very core of the information-mass nexus.
“One of the most heavily quoted passages in physics”
I’ve always been fascinated and intrigued by the connections between information, computing, and physics. Indeed, during the first two years of my physics BSc at Dublin City University it was not too infrequently that I found myself thinking that I should have done a computer science degree instead. (I’ve never been the best of mathematicians but I was a reasonable coder; discrete and numerical methods always “clicked” a little more with me than analytical maths. My mantra throughout my undergrad degree was “If I can’t see how to code this, I don’t understand it”).
I pop into DCU any time I’m in Dublin and on one of those visits I spotted the book below on a friend’s bookshelves and asked him whether I could borrow it. It’s a real gem, which I recommend to anyone with even a passing interest in the intriguing and multi-facetted role that information plays in physics. (Tony, if you’re reading, I am hugely sorry that I’ve held onto the book for so long. I’ll return it next time I’m back home – promise!)
The contents of, and motivation for, this captivating book are best described by the blurb on its back cover:
About 120 years ago, James Clerk Maxwell introduced his now legendary hypothetical ‘demon’ as a challenge to the integrity of the second law of thermodynamics. Fascination with the demon persisted throughout the development of statistical and quantum physics, information theory and computer science – and links have been established between Maxwell’s demon and each of those disicplines. The demon’s seductive quality makes it appealing to physical scientists, engineer, computer scientists, biologists, psychologists, and historians and philosophers of science.
“Maxwell’s Demon: Entropy, Information, Computing” is a collection of twenty-five reprints on the subject of Maxwell’s demon (and related themes) prefaced by an engaging overview by Harvey Leff and Andrew Rex that synopsises the key developments in our understanding of the links between information, entropy, energy, and computing stimulated by that eponymous beast.
The demon was birthed by Maxwell in his Theory of Heat (1871) and “in one of the most heavily quoted passages in physics”, as Leff and Rex put it, described thus:
The Sixty Symbols video embedded above describes how the demon works (with my daughter Saoirse’s Living Dead Doll assuming the role of the fiend) but Maxwell’s pithy description above tells you all you need to know in any case. The demon keeps a careful eye on molecules in a box which is separated into two chambers by a partition/door. He/she/it opens a door to allow fast-moving molecules to pass into chamber B, while those moving more slowly are allowed to pass to chamber A. The key point is as I’ve underlined above: the demon works without expending work, establishing a temperature difference that could potentially be exploited, and thus the second law of thermodynamics is violated. (More on work, in the physics sense of the word, below).
I’ll note in passing that Maxwell’s careful qualifier re. the faculties of the demon, i.e. “would be able to do what is at present impossible to us”, is remarkably prescient in the context of the invention of scanning probe microscopy (SPM) about a century after the Theory of Heat was published. Probe microscopes now routinely allow us to not only see individual atoms and molecules but to manipulate them one at a time, and the state of the art in the field involves resolving the internal bond architecture of single molecules. (It is also worth comparing and contrasting Maxwell’s considered use of the “at present” proviso with Schrodinger’s rather more gung-ho statement in 1952: “We never experiment with just one electron or atom or (small) molecule. In thought experiments we sometimes assume that we do; this invariably entails ridiculous consequences…In the first place it is fair to say that we cannot experiment with single particles, any more than we can raise ichtkyosauria in the zoo”)
This version of Maxwell’s demon, which sets up a temperature gradient in a gas of molecules, is but one of a family of little devils. Maxwell went on to envisage a rather more stupid demon which didn’t need to keep account of molecular speeds, but instead simply opened the partition for molecules travelling one way and not the other. As Maxwell put it (p. 6 of the 1st edition of Leff and Rex’s book): “This reduces the demon to a valve”.
Make everything as simple as possible, but no simpler
It wasn’t, however, until Leo Szilard introduced the “spherical cow” version of the demon in 1929 that the links between information, entropy, and energy started to become clear. Physicists love to reduce a system down to its barest bones; some of us are rather simple-minded beasts so we prefer to cut out any extraneous complexity and get to the heart of the matter. Szilard got rid of all of the molecules in the demon’s purview…save for one, lonely particle. In other words, he considered a single molecule gas. (Another note in passing: I made this spherical cow point in the Sixty Symbols video only to subsequently find that Sean Carroll also includes mention of the bulbous bovine in his wonderfully clear and pithy description of Szilard’s demon here. I thoroughly recommend Carroll’s blog and books. He’s a fantastic science communicator, as his videos for Sixty Symbols highlight very well. (I can’t say, however, that I share Sean’s unalloyed enthusiasm for the many-worlds interpretation of quantum mechanics.))
Szilard reduces the information overload of the original Maxwellian demon to a very simple problem for his incarnation of the devil: which side of the container is the molecule on? As described in the video, if Szilard’s (rather lazier) demon knows on which side of the container the molecule is found then work can be extracted, without having to put any work into the system in the first place. Another free lunch.
Ultimately, and after decades of debate, these violations of the 2nd law were traced back to an aspect of the problem that was too often overlooked: the information that the demon, of whatever type, has acquired. In Szilard’s case, this is one bit of information: what side of the container is the molecule on? It’s a binary problem.
A bit of energy
What’s great is that the simplicity of Szilard’s model means that we can use 1st year thermodynamics (or A-level thermal physics) to work out a formula for the energy (or, alternatively, entropy) associated with this single bit of information. In the video I simply write this formula down (Ebit = kT ln 2) but we can derive it in just a few lines…
The infinitesimal  amount of work, dW, done by a gas on a piston — in other words, the infinitesimal amount of energy extracted from an expanding gas — is given by
dW = PdV
where P is the pressure of the gas and dV is the change in volume of the gas. 
For Szilard’s demon the volume occupied by the ‘gas’ (i.e. the single molecule) changes by a factor of 2 as it expands: the demon observes which half of the box contains the molecule and acts accordingly. The volume changes from Vbox/2 to Vbox as the single molecule gas expands, pushing back the piston.
Now, if we want to determine the total work done by the molecule during this process then we integrate up all those infinitesimal “chunks” of work within the limits of Vbox and Vbox/2:
Fine, you might say, but how can we do the integration if we don’t know how the pressure is related to the volume? Not a problem. We do know how the pressure and volume are related. It’s the ideal gas law you may have learned in secondary/high school science classes,
PV = nRT
Here, P and V are once again pressure and volume, R is the universal gas constant, T is temperature, and n is the number of moles of gas.
But we’re only dealing with one molecule for Szilard’s engine so the ideal gas law is even simpler. We don’t need to worry about moles, so we don’t need the universal gas constant, and we can instead write for a single molecule:
The k in that equation is Boltzmann’s constant – it’s the universal conversion factor between energy and temperature. 
We now have an expression for P in terms of V, namely P = kT/V
Let’s plug that into the integral above:
Now, kT is a constant (because the temperature is constant in Szilard’s model). That means we can take it out of the integral, like this:
The integral of 1/V is ln V (i.e. the natural log of V). If we evaluate that integral between the given limits then we get the following:
But in “Logland” subtraction is equivalent to division of the arguments, so we have:
W = kT ln 2
And there’s our formula for the energy associated with a single bit of information. (In terms of entropy, the formula for one bit is even simpler still: S = k ln 2).
(There have, of course, been objections to the type of reasoning above. (Here’s one example from my fellow scanning probe microscopist, Quanmin Guo). Leff and Rex’s book details the objections and describes how they were addressed.
It from bit
In the video, Brady and I – with tongues very firmly in cheeks – consider the energy content of a “thought” (and those scare quotes are very important indeed): a simple image, whose total number of bits can be determined from the pixel density, assuming 24 bits per pixel. We then compare that “information energy” with the nutritional energy value of a Mars bar. 
I can already hear the disgruntlement of certain factions complaining about “dumbing down” and “clickbait”  but Sixty Symbols videos were never meant to be tutorials – they’re about piquing interest. If someone (anyone!) comes away from the video thinking, like I do, “Wow, those links between information, energy, and entropy are fascinating. I’d like to find out more”, then I consider that to be job done.
In any case, to begin to do justice to the topic would require a lengthy series of videos (or a 30-hour-long single video). (Or, alternatively, those interested could read Leff and Rex’s book.) But, Brady willing, we’ll hopefully return in a Sixty Symbols video some time to a consideration of Wheeler’s famous “It from bit” statement, Landauer’s mantra of “Information is physical”, and the central importance of data erasure. On this latter point, it turns out that what’s really important is not storing information, but erasing/forgetting it. The demon needs to be just like a stereotypical physics professor: absent-minded.
And if you’ve made it this far in this long-winded post, I think you’d agree that it’s now a case of too much information…
 Smaller than the smallest thing ever, and then some. (Hat tip to Mr. Adams). James Grime did an engaging video on infinitesimals for Brady’s Numberphile channel.
 For the experts among you, yes we should be careful to note when we have an exact vs inexact differential; and, yes, we should be careful with + and – signs regarding the representation of whether work is done on, or by, the gas; and, yes, we should also in principle take care to explain the difference between irreversible and reversible processes. I know. Let it go. The goal here is to put across a broad concept to a broad audience, and the minutiae don’t matter when explaining that concept. 
 Tetchy? Me?
 If you’re wondering how we replaced R with k, note that R = NAk, where NA is Avogadro’s constant. In other words, R, the universal gas constant, is a “mole-full” of Boltzmann’s constants.
 Some have gone further and used E=mc2 to assign a mass to a bit of information. In that sense, we could even ask what’s the weight of a thought. We didn’t want to do this, however, because explaining mass-energy equivalence correctly requires a great deal of care, and the video was already too long to include that type of nuance.