The power of thought?

Ponder this…

How much energy is associated with information? With knowledge? With thought?

Could we extract power solely from information, from a state of knowing?

In other words, is there a thermodynamics of information? (Don’t click on that link just yet if you want to avoid spoilers..)

These are the types of weighty question Brady Haran and I explored in the most recent video for Sixty Symbols, which Brady uploaded today.

Each time I receive the e-mail from Brady telling me he’s uploaded a video to which I’ve contributed, I get that familiar feeling in the pit of my stomach stemming from the worry that I might have screwed up the explanation of the physics and potentially misled those who are watching.

In a short(ish) video, we necessarily have to gloss over and/or omit lots of detail and, as a physics lecturer, this is always uncomfortable. (Deeply, deeply uncomfortable). Indeed, and as I described in a Physics World article a couple of years ago, for some time I decided to withdraw from making Sixty Symbols videos for precisely this reason. But as also discussed in that article, the ability to connect with an audience who are enthralled by, and enthusiastic about, physics, the intellectual challenge of explaining difficult concepts, and the sheer fun of working with Brady (our occasional spats notwithstanding) meant that I quickly saw the error of my ways. (As, of course, Mr Haran had predicted.)

Nonetheless, if there’s one topic that I find exceptionally difficult to put across in the short, snappy, “bite-size” YouTube format, it’s entropy. I had sworn off trying to explain the intricacies off this particular thorny concept in the Sixty Symbols style, but I keep getting drawn back to it, almost against my will — because it’s so damn interesting. When it came to thinking about thoughts, entropy and energy, we had to bite the bullet because entropy is at the very core of the information-mass nexus.

“One of the most heavily quoted passages in physics”

I’ve always been fascinated and intrigued by the connections between information, computing, and physics. Indeed, during the first two years of my physics BSc at Dublin City University it was not too infrequently that I found myself thinking that I should have done a computer science degree instead. (I’ve never been the best of mathematicians but I was a reasonable coder; discrete and numerical methods always “clicked” a little more with me than analytical maths. My mantra throughout my undergrad degree was “If I can’t see how to code this, I don’t understand it”).

I pop into DCU any time I’m in Dublin and on one of those visits I spotted the book below on a friend’s bookshelves and asked him whether I could borrow it. It’s a real gem, which I recommend to anyone with even a passing interest in the intriguing and multi-facetted role that information plays in physics. (Tony, if you’re reading, I am hugely sorry that I’ve held onto the book for so long. I’ll return it next time I’m back home – promise!)

879150.jpg

The contents of, and motivation for, this captivating book are best described by the blurb on its back cover:

About 120 years ago, James Clerk Maxwell introduced his now legendary hypothetical ‘demon’ as a challenge to the integrity of the second law of thermodynamics. Fascination with the demon persisted throughout the development of statistical and quantum physics, information theory and computer science – and links have been established between Maxwell’s demon and each of those disicplines. The demon’s seductive quality makes it appealing to physical scientists, engineer, computer scientists, biologists, psychologists, and historians and philosophers of science.

Maxwell’s Demon: Entropy, Information, Computing” is a collection of twenty-five reprints on the subject of Maxwell’s demon (and related themes) prefaced by an engaging overview by Harvey Leff and Andrew Rex that synopsises the key developments in our understanding of the links between information, entropy, energy, and computing stimulated by that eponymous beast.

The demon was birthed by Maxwell in his Theory of Heat (1871) and “in one of the most heavily quoted passages in physics”, as Leff and Rex put it, described thus:

 

The Sixty Symbols video embedded above describes how the demon works (with my daughter Saoirse’s Living Dead Doll assuming the role of the fiend) but Maxwell’s pithy description above tells you all you need to know in any case. The demon keeps a careful eye on molecules in a box which is separated into two chambers by a partition/door. He/she/it opens a door to allow fast-moving molecules to pass into chamber B, while those moving more slowly are allowed to pass to chamber A. The key point is as I’ve underlined above: the demon works without expending work, establishing a temperature difference that could potentially be exploited, and thus the second law of thermodynamics is violated. (More on work, in the physics sense of the word, below).

I’ll note in passing that Maxwell’s careful qualifier re. the faculties of the demon, i.e. “would be able to do what is at present impossible to us”, is remarkably prescient in the context of the invention of scanning probe microscopy (SPM) about a century after the Theory of Heat was published. Probe microscopes now routinely allow us to not only see individual atoms and molecules but to manipulate them one at a time, and the state of the art in the field involves resolving the internal bond architecture of single molecules. (It is also worth comparing and contrasting Maxwell’s considered use of the “at present” proviso with Schrodinger’s rather more gung-ho statement in 1952: “We never experiment with just one electron or atom or (small) molecule. In thought experiments we sometimes assume that we do; this invariably entails ridiculous consequences…In the first place it is fair to say that we cannot experiment with single particles, any more than we can raise ichtkyosauria in the zoo”)

This version of Maxwell’s demon, which sets up a temperature gradient in a gas of molecules, is but one of a family of little devils. Maxwell went on to envisage a rather more stupid demon which didn’t need to keep account of molecular speeds, but instead simply opened the partition for molecules travelling one way and not the other. As Maxwell put it (p. 6 of the 1st edition of Leff and Rex’s book): “This reduces the demon to a valve”.

Make everything as simple as possible, but no simpler

It wasn’t, however, until Leo Szilard introduced the “spherical cow” version of the demon in 1929 that the links between information, entropy, and energy started to become clear. Physicists love to reduce a system down to its barest bones; some of us are rather simple-minded beasts so we prefer to cut out any extraneous complexity and get to the heart of the matter. Szilard got rid of all of the molecules in the demon’s purview…save for one, lonely particle.  In other words, he considered a single molecule gas.  (Another note in passing: I made this spherical cow point in the Sixty Symbols video only to subsequently find that Sean Carroll also includes mention of the bulbous bovine in his wonderfully clear and pithy description of Szilard’s demon here.  I thoroughly recommend Carroll’s blog and books. He’s a fantastic science communicator, as his videos for Sixty Symbols highlight very well. (I can’t say, however, that I share Sean’s unalloyed enthusiasm for the many-worlds interpretation of quantum mechanics.))

Szilard reduces the information overload of the original Maxwellian demon to a very simple problem for his incarnation of the devil: which side of the container is the molecule on? As described in the video, if Szilard’s (rather lazier) demon knows on which side of the container the molecule is found then work can be extracted, without having to put any work into the system in the first place. Another free lunch.

Ultimately, and after decades of debate, these violations of the 2nd law were traced back to an aspect of the problem that was too often overlooked: the information that the demon, of whatever type, has acquired. In Szilard’s case, this is one bit of information: what side of the container is the molecule on? It’s a binary problem.

A bit of energy

What’s great is that the simplicity of Szilard’s model means that we can use 1st year thermodynamics (or A-level thermal physics) to work out a formula for the energy (or, alternatively, entropy) associated with this single bit of information. In the video I simply write this formula down (Ebit = kT ln 2) but we can derive it in just a few lines…

The infinitesimal [1] amount of work, dW, done by a gas on a piston — in other words, the infinitesimal amount of energy extracted from an expanding gas — is given by

dW = PdV

where P is the pressure of the gas and dV is the change in volume of the gas. [2]

For Szilard’s demon the volume occupied by the ‘gas’ (i.e. the single molecule) changes by a factor of 2 as it expands: the demon observes which half of the box contains the molecule and acts accordingly. The volume changes from Vbox/2 to Vbox as the single molecule gas expands, pushing back the piston.

Now, if we want to determine the total work done by the molecule during this process then we integrate up all those infinitesimal “chunks” of work within the limits of Vbox and Vbox/2:

integral1

Fine, you might say, but how can we do the integration if we don’t know how the pressure is related to the volume? Not a problem. We do know how the pressure and volume are related. It’s the ideal gas law you may have learned in secondary/high school science classes,

PV = nRT

Here, P and V are once again pressure and volume, R is the universal gas constant, T is temperature, and n is the number of moles of gas.

But we’re only dealing with one molecule for Szilard’s engine so the ideal gas law is even simpler. We don’t need to worry about moles, so we don’t need the universal gas constant, and we can instead write for a single molecule:

PV=kT

The k in that equation is Boltzmann’s constant – it’s the universal conversion factor between energy and temperature. [4]

We now have an expression for P in terms of V, namely P = kT/V

Let’s plug that into the integral above:

integral2

Now, kT is a constant (because the temperature is constant in Szilard’s model). That means we can take it out of the integral, like this:

integral3

The integral of 1/V is ln V (i.e. the natural log of V). If we evaluate that integral between the given limits then we get the following:

integral4

But in “Logland” subtraction is equivalent to division of the arguments, so we have:

W = kT ln 2

And there’s our formula for the energy associated with a single bit of information. (In terms of entropy, the formula for one bit is even simpler still: S = k ln 2).

(There have, of course, been objections to the type of reasoning above. (Here’s one example from my fellow scanning probe microscopist, Quanmin Guo). Leff and Rex’s book details the objections and describes how they were addressed.

It from bit

In the video, Brady and I – with tongues very firmly in cheeks – consider the energy content of a “thought” (and those scare quotes are very important indeed): a simple image, whose total number of bits can be determined from the pixel density, assuming 24 bits per pixel. We then compare that “information energy” with the nutritional energy value of a Mars bar. [5]

I can already hear the disgruntlement of certain factions complaining about “dumbing down” and “clickbait” [3] but Sixty Symbols videos were never meant to be tutorials – they’re about piquing interest. If someone (anyone!) comes away from the video thinking, like I do, “Wow, those links between information, energy, and entropy are fascinating. I’d like to find out more”, then I consider that to be job done.

In any case, to begin to do justice to the topic would require a lengthy series of videos (or a 30-hour-long single video). (Or, alternatively, those interested could read Leff and Rex’s book.) But, Brady willing, we’ll hopefully return in a Sixty Symbols video some time to a consideration of Wheeler’s famous “It from bit” statement, Landauer’s mantra of “Information is physical”, and the central importance of data erasure. On this latter point, it turns out that what’s really important is not storing information, but erasing/forgetting it. The demon needs to be just like a stereotypical physics professor: absent-minded.

And if you’ve made it this far in this long-winded post, I think you’d agree that it’s now a case of too much information…


 

[1] Smaller than the smallest thing ever, and then some. (Hat tip to Mr. Adams). James Grime did an engaging video on infinitesimals for Brady’s Numberphile channel.

[2] For the experts among you, yes we should be careful to note when we have an exact vs inexact differential; and, yes, we should be careful with + and – signs regarding the representation of whether work is done on, or by, the gas; and, yes, we should also in principle take care to explain the difference between irreversible and reversible processes. I know. Let it go. The goal here is to put across a broad concept to a broad audience, and the minutiae don’t matter when explaining that concept. [3]

[3] Tetchy? Me?

[4] If you’re wondering how we replaced R with k, note that R = NAk, where NA is Avogadro’s constant. In other words, R, the universal gas constant, is a “mole-full” of Boltzmann’s constants.

[5] Some have gone further and used E=mc2 to assign a mass to a bit of information. In that sense, we could even ask what’s the weight of a thought. We didn’t want to do this, however, because explaining mass-energy equivalence correctly requires a great deal of care, and the video was already too long to include that type of nuance.

 

 

Author: Philip Moriarty

Physicist. Rush fan. Father of three. (Not Rush fans. Yet.) Rants not restricted to the key of E minor...

24 thoughts on “The power of thought?”

      1. Sorry if I’m spamming your comment section. I needed a break from what I’m supposed to be doing this morning and you’re it I’m afraid.

        The comment you make around 11 minutes in, that ‘any substantial leap in my understanding has not come when I’ve felt smart but when I’ve felt stupid. Really stupid’, struck a chord with me.

        It puts me in mind of a PhD thesis written by Derek Muller (Veritasium on Youtube) which talks about tests he ran on the success of two different videos in allowing students to understand (I think) Newton’s second law. One video simply presented the facts as Newton had laid them out whilst the second illustrated the common, but incorrect, intuitions we have about the ways object interact, followed by a correction.

        Students who watched the first video apparently felt that they unproblematically understood it and were ‘smarter’ as a result of having watched it. Those who watched the second were, however, baffled by it; they found it harder and felt consequently stupider.

        Interestingly though, when tested afterwards it was the group who had suffered through the second video who had a better grasp of what the law actually stated. It seems that the first group, lulled by their own feeling of smartness, had not actually engaged with the content and how it might contradict their intuitions.

        When I was in education it was that kind of reckoning which I found most inspiring, and the most interesting challenge when introducing students to a new subject; how to meet them where they are but also to provide a vehicle for taking them to the place they needed to be, even if that meant rough going at times.

        As you say though, one of the problems of trying this is the modern TEF-minded university, is that quality teaching can be sacrificed in favour of banal student satisfaction measures which reward the production of feeling smart.

        Best wishes

        Fred

        Like

  1. Hello prof. i`ve seen you at Sixty Symblos, hopefully mby it`ll be easier to get some answers in here 🙂 I have some ideas about the topic of Maxwell demon.
    How if there is conservation of energy that says smth like:
    “Energy required to process infromation about process can`t be lesser than energy that can be extracted from this process”.
    For example 1 dimension Deamon can`t extract work from 1 atom because he/she still need to monitor atom side (piece of cake) and ALSO velocity to get timing of piston right.
    Higher velocity need faster data processing and this requires more energy to procces information on time.
    I`ve done that post and i ve realised tht we can increase energy by increasing mass (so the date processing doesn`t need to be faster)…
    Damn you Deamon ;D!

    Like

    1. Hi, Grzegorz.

      I’m not certain that I entirely follow your comment but I can assure you that the 1st law of thermodynamics (i.e. the conservation of energy) holds in Szilard’s model. The expansion of the single molecule “gas” is an isothermal process — the temperature of the gas remains constant.

      Philip

      Like

  2. A question on self fulfilling prophecies.

    If an equation is written that heat is generated in a process and “laws” are written like the thermodynamic laws, is it not possible that things are invented to fit the narrative? I mean does physicists and engineers unconsciously create machines that create heat as a byproduct just because their minds are hallucinating that heat must be produced, but as little heat as possible (efficiency) which impresses the peers.

    Perhaps it is not necessary, but our limited understanding and rudimentary observations since the age of steam engines and burning trees.

    It is not a scientific question, just something to throw doubts into the thinking. Like the devils advocate.

    Take the story about the Japanese manufacturing wonder. An European buyer of automotive parts ordered 95 % working parts and 5 % non working parts (which were the norm of the quality in the manufacturing in Europe. The Japanese supplier sent a separate box of non working parts with a note why the buyer wanted non working parts. The Japanese had only working parts in mind and therefore were clueless to why someone would like non working parts.

    Like

    1. If an equation is written that heat is generated in a process and “laws” are written like the thermodynamic laws, is it not possible that things are invented to fit the narrative?

      What a fantastic question!

      Yes, there is of course some degree of “fitting the narrative” as you put it, in science. But the key points are that (a) we are always (or should always be!) constrained by the evidence, and (b) (and this is really key) a model/equation/theory should have predictive power. It should predict the outcomes of experiments we haven’t yet done so that we can check its validity and be rather more confident we’re not fooling ourselves.

      And if our theory/model is not supported by evidence and/or make quantifiably verifiable predictions then we have much less confidence in it. (It’s why I twitch a little when economics is described as a science. Where are the quantifiable predictions?)

      Unfortunately, some scientists have suggested that because acquiring evidence is getting increasingly difficult we should move to an era of post-empirical science. Ugh. See https://muircheartblog.wordpress.com/2015/08/25/when-scientists-help-to-sell-pseudoscience-the-many-worlds-of-woo/

      Thanks for your thought-provoking comment.

      Philip

      Like

  3. I’m not sure I can unpick the video, but a while ago Jim al-Khalili mentioned Maxwell’s Demon in one of his BBC programmes.

    At the time it got me thinking, given (what I understand to be) the interconnectedness of the mind and the system it is observing there seem to be obvious parallels with the double-slit experiment.

    Can we apply lessons from Maxwell’s Demon to the double-slit?

    That would seem like fertile ground for research. Is it?

    Like

  4. Since my knowledge of maths/physics is √-1 this is probably a dumb thought but … since a ‘thought’ is a bit different from a picture (unless you’re a follower of the early Wittgenstein I guess) wouldn’t it be better to plug the data about the energy content of a Mars bar into a more relevant equation? I’d vote for Tononi’s equations that specify his ‘integrated information theory’.

    Regards

    Fred

    Like

    1. Hi, Fred.

      It’s really not meant to be taken too literally! The key concept here is the relationship between information, entropy, and energy.

      “d vote for Tononi’s equations that specify his ‘integrated information theory’.”

      But this would over-complicate matters hugely. The purpose of the video, like any Sixty Symbols video, is to pique interest. That’s all. It’s not a tutorial. It’s not a comprehensive analysis. It’s not an academic paper. I simply want those who watch the video to go “Oh, that’s interesting. I’d like to find out more about that”.

      Moreover, there is huge value in gedankenexperiments that strip out all extraneous detail and focus on the core physics. That’s why Szilard’s model is so attractive. He worked out the energy (and entropy) of a bit of information. This, of course, like all models in reversible, equilibrium thermodynamics is a huge idealisation. But like many idealisations in physics, it provides a great deal of conceptual insight.

      For example, when teaching the physics of what happens when an apple falls to the ground our starting point is not to include the fluid dynamics of the airflow round the apple. Indeed, we ignore air resistance…

      Like

      1. Hi Phil,

        Taking metaphors too literally is kinda my thing I’m afraid, as well as noting when and for what purpose literal things get turned into metaphors.

        Really enjoyed the video though, as always (although my personal favourite in terms of metaphors and their uses/limits, was one from a few years back about ‘Do atoms touch’.

        Best wishes

        Fred

        Liked by 1 person

    2. Hi, Fred.

      Delighted to hear that you enjoyed the video. Thanks for that!

      The comments formatting in this WordPress theme is really not the best. Only about three “layers” of embedding is allowed. Now, I could look into fixing this but I really don’t have the time at the moment to dig into coding style sheets and the like so I’m afraid we’ll have to live with it.

      So, although it seems like I’m responding to your comment re metaphors, it’s actually this I’m responding to:

      It puts me in mind of a PhD thesis written by Derek Muller (Veritasium on Youtube) which talks about tests he ran on the success of two different videos in allowing students to understand (I think) Newton’s second law. One video simply presented the facts as Newton had laid them out whilst the second illustrated the common, but incorrect, intuitions we have about the ways object interact, followed by a correction.

      Thanks for raising this really relevant point. I wrote a post a while ago on the value of confusion in teaching which refers to Derek’s work: https://muircheart.wordpress.com/2014/11/06/moriarty-confused-good-learn/

      Best wishes,

      Philip

      Like

  5. Hi Professor,

    It seems to me that Szilard thought experiment allows us to find the energy value of one specific bit of information in a given context (namely that of the thought experiment) – what allows you to extrapolate this to say that this is the value of a bit of information in the general case? I appreciate that it may be to do with the fact that it is a binary true/false claim, but surely this is only one shared feature with bits of information and it is easy to imagine that in a different context the energy value may be different.

    Also, how does this actually solve the 2nd Law problem? Yes we have now assigned a value to this bit of information, but that doesn’t change the fact that no energy is physically put into the system by the moving of the barrier in either version of the thought experiment.

    Like

    1. Hi.

      It solves the 2nd law law problem because we’re no longer getting work out of nowhere. There’s an entropy related to the information that the demon acquires. (Well, actually, what’s more important is when that information is erased, as Landauer showed).

      “but that doesn’t change the fact that no energy is physically put into the system by the moving of the barrier “

      It depends on what you mean by this. The single molecule gas expands isothermally in both cases of the demon I discuss above. This means its internal energy remains constant. Energy conservation holds because the container is coupled to a heat bath.

      The petition/valve is frictionless. Therefore, in the absence of any interaction with the molecule there is no energy cost — there is no force against which the barrier has to do work.

      Philip

      Like

  6. @Fred.

    I’m really pleased to hear that you like the “Do atoms touch?” video. I really like the exchange Brady and I had in that video — it shows science as a debate/an exchange rather than a static “repository” of facts.

    Thanks for commenting.

    Philip

    Like

  7. Prof.Moriarty,
    I’m a physics student and i have the following question involving Q.M:

    What if we had 2 quantum objects with a barrier between them
    and the box was only 3 ‘object size’ wide (maybe Planck length?)
    and then random quantum tunneling happened, moving the object
    to the other site and creating work, without the object,
    after that, ba able to return back (and restore equilibrium),
    because the bar is part of the infinite wall now?

    Does it make any sense, or the thought process is wrong?*
    *Maybe they only can move vertically.

    xxxxxxxxxx xxxxxxxxxx
    xxxxxxxxxx xxxxxxxxxx
    xxx Ioxxxx xxxo Ixxxx
    xxx I xxxx == | xxx Ixxxx
    xxxoI xxxx / xxxo Ixxxx
    xxxxxxxxxx xxxxxxxxxx
    xxxxxxxxxx xxxxxxxxxx

    P.S
    I really like the video of you explaining “Why is glass transparent?”
    and is one of the reasons I chose to study Physics. Thanks!

    Like

  8. Hi phill
    A quick question, I kind of know the answer but I would like you views. Does it matter what the picture is of. For example a picture of a random static is has many more micro-states that say a picture of chess board. So does the chess board have less entropy than a random picture and then can I have more ordered thoughts in my mars bar?
    cheers

    Like

  9. Hey Phil, your derivation of the energy (entropy?) of 1 bit of information made that part of the video a little clearer, thanks. Now how does this relate back to the demon? Do we assume the demon acquires this knowledge in some frictionless fashion but that it’s energy is opposite in sign, thus cancelling out the work that we can do with the piston and preserving the 2nd law? If so, why is the energy of a bit “negative”? The video and your post seemed to gloss over that part of the solution to Maxwell’s demon imho.

    Like

Comments are closed.