Recode Reality
Recode Reality Ten Thousand Things

Inference Machine

推理 Inference Machine

Open your eyes and the world appears at once. This is the trick. The world was already there before you opened them — in the brain, as a prediction the eyes are about to confirm or correct.

The lifting of the eyelids does not deliver the room to you. The room is constructed faster than the lifting takes — assembled by a process that has been running continuously beneath your awareness, drawing on every previous time you have been in a room shaped like this one, with light angled like this, with furniture in roughly these positions. What the eyes contribute, when they open, is not the room. The room was already there. The eyes contribute corrections.

This is the harder claim. Perception is not the senses delivering the world to a passive observer. Perception is the brain generating the world, moment by moment, and the senses checking the brain's work.

· · ·

There is a small region in each of your eyes where no light is sensed. Where the optic nerve leaves the back of the eye and travels to the brain, there is a hole in the retina — a circle of about two millimetres in diameter where there are simply no photoreceptors. Anything whose image falls on this region is invisible to you. You can prove this to yourself in thirty seconds. Close one eye. Hold a finger out at arm's length. Slowly move it sideways while staring fixedly ahead. At a specific spot, the fingertip will vanish. It is not faint. It is not blurred. It is gone. The blind spot has eaten it.

This region is in your visual field every moment of your waking life. It is not small in apparent size — at arm's length, it is about the diameter of a tennis ball. And yet you do not see a hole there. You do not see a black disc floating in your peripheral vision. You see whatever happens to be around the blind spot, extended smoothly across the missing region. If the background is sky, the brain fills the blind spot with sky. If the background is striped wallpaper, the brain fills it with stripes that continue the pattern. The fingertip vanishes because there is no signal from that location — and the brain replaces the missing signal with what would most plausibly be there.

The brain is not failing to notice the hole. The brain is generating the missing patch.

This is the most dramatic version of something the brain is doing all the time, in every region of the visual field, with every sense. You can demonstrate it more subtly with a familiar effect. You are at a party. The room is full of conversations — twenty, thirty voices overlapping, none of them clearly audible above the others. You are talking to someone in your own conversation, focused on what they are saying. Then, somewhere across the room, in a conversation you have not been listening to, someone says your name. You hear it instantly. Your attention turns. The voice that said your name was no louder than any of the other voices in the room. It crossed the same acoustic threshold as every other word being said in that conversation. But your name reached you in a way that the surrounding words did not.

The brain was not, in fact, ignoring the other conversations. The brain was processing all of them — running each conversation against a list of things it was watching for, your name among them. Most words failed to match anything important and were filtered out before reaching your awareness. Your name matched something — a prediction maintained at high precision, ready to be confirmed at any moment — and crossed into awareness immediately. The signal that surfaced was not the loudest signal. The signal that surfaced was the most highly predicted one.

A third example, more carefully constructed. In 1970, the American psychologist Richard Warren ran a now-classic experiment. He played his subjects recorded sentences in which one phoneme had been removed and replaced with a cough — a noise that completely masked the missing speech sound. The sentence was a context that could be completed by several different words. The state governors met with their respective legi[cough]atures convening in the capital city. Subjects heard legislatures, and they heard the cough as separate, occurring underneath or alongside the word — not in place of the missing phoneme. The brain reconstructed the missing sound from context. Warren ran variations: change the surrounding sentence, and the same coughed-over gap was reconstructed as different words — wheel, peel, heel, meal — depending on what the sentence required. The phoneme was not heard because it was there. The phoneme was heard because the sentence predicted it.

A fourth example, the strangest. In 1976, the British psychologist Harry McGurk and his collaborator John MacDonald accidentally discovered something while preparing video stimuli for an unrelated experiment. They were dubbing audio recordings of one syllable onto video of someone speaking a different syllable. When they watched the result — a face whose lips clearly mouthed "ga" while the audio track played "ba" — they did not hear "ba" and they did not hear "ga". They heard a third syllable: "da". A fusion that matched neither input. McGurk and MacDonald could not at first believe what they were experiencing; they thought they had labelled the tapes wrongly. They had not. Cover your eyes while the audio plays and "ba" returns instantly. Watch the lips again and "da" comes back. The brain was using the visual information about the lip movements to predict what the auditory signal should be, and when the auditory signal failed to match the visual prediction, the brain split the difference. What you hear is not what your ears delivered. It is what your brain calculated should be heard, given everything coming in.

What runs through all four of these examples is a consistent pattern. The brain is not waiting for the senses to deliver information and then processing it. The brain is predicting, in advance, what the senses ought to be delivering — and the experience that surfaces is the prediction, edited where the sensory data forces an edit. The blind spot is filled because the prediction fills it. Your name crosses the noise because the prediction is maintained for it. The missing phoneme is heard because the prediction inserts it. The McGurk syllable is heard because the prediction fuses the two channels into something that matches neither.

This is not a quirk of perception under unusual conditions. This is how perception works under all conditions. The model is the experience. The senses are what edit the experience when the model fails.

Perception is not the brain receiving the world. Perception is the brain generating its best guess at the world, then letting the senses correct the worst of the errors.
· · ·

Imagine you are listening to a song you have heard many times. Hey Jude, perhaps — the long fade-out, the na na na na chorus you can sing from memory. As the song plays, your brain is doing something specific that has been invisible to you because it is doing it well. At every moment, before each note arrives, the brain is predicting what the next note will be. As each note actually arrives, it confirms the prediction. The note that was expected is the note that arrived. Upstairs in the brain, nothing happens. Everything is quiet because nothing is surprising.

Now the same song with an unexpected chord change. The surface layer of the brain — the part holding the most concrete, fastest-changing prediction, the prediction of the next note — has its prediction fail. A small signal travels upward, into the layer above. That layer holds a slightly more abstract prediction: the prediction of the next musical phrase, the expectation of where the melody is going. This layer adjusts. If the unexpected chord change is small enough to fit within the phrase the song is in, the adjustment ends here. If the chord change is bigger than that — if it breaks the phrase entirely — the error signal travels further up, into the layer holding the prediction this is Hey Jude. That layer reconsiders. With enough wrong notes, the deep layer updates: this is not Hey Jude. This is a cover. This is a different song entirely. And as soon as the deep layer updates, the layers below it recalibrate too — they now expect the notes of the new song, and the brain goes quiet again until the next surprise arrives.

The crucial point is what travels upward through this hierarchy. It is not the music. The music is not being passed up the brain, layer by layer, like a packet being relayed through a sequence of routers. What travels upward is only the part of the music that the lower layer did not predict. Prediction errors. Residual surprise. If every note is predicted correctly, nothing travels upward at all. The hierarchy is quiet because there is nothing to report. Information only moves through the brain when something fails to match expectation.

This mechanism has a technical name. It is called predictive coding. The modern form of the theory was proposed in 1999 by Rajesh Rao and Dana Ballard, looking at the visual cortex — though it has older roots in the work of Hermann von Helmholtz in the 1860s. Rao and Ballard showed that the response patterns of neurons in the visual cortex match the predictions of a model in which each layer is sending predictions downward to the layer below it, and the layer below is sending prediction errors upward. In the years since, the same architecture has been found, with variations, across the cortex. The framework has expanded into a general theory of how the brain works — variously called predictive processing, the Bayesian brain, or active inference, depending on which aspect is being emphasised. They are all describing the same observation: the brain is not passively receiving information from the world. The brain is actively predicting the world, and using the senses to correct itself where it predicts wrong.

There is one further move that needs naming, because it is consequential.

So far, prediction has been described as something the brain does to perceive. But prediction does more than this. Prediction also acts. Imagine your brain expects the cup of coffee to be on the left side of your desk. Your eyes report that the cup is on the right. There is a mismatch — a prediction error. The brain has two ways to resolve it. It can update the prediction (the cup is on the right). Or it can update the world (reach out with the right hand and move the cup to the left). Both reduce the prediction error. Both are forms of the same process.

This is active inference — the framework Karl Friston extended predictive processing into, beginning in the early 2000s. The organism does not separate perception from action. Both are doing the same fundamental job — closing the gap between what is predicted and what is actually happening. Perception updates the prediction to match the world. Action updates the world to match the prediction. They are two outputs of one underlying process.

When you reach for a cup, the reach is not the consequence of first perceiving the cup and then deciding to reach. The reach is itself a prediction — my hand will be at the cup — that the motor system makes true. The brain's expectations are not just descriptions of what is. They are also descriptions of what is about to be, with the body's actions filling in the difference.

The Edinburgh philosopher Andy Clark, who has spent decades thinking and writing about this, has a single line that captures the move. Human brains are prediction machines, he writes in his 2023 book The Experience Machine. The phrasing is deliberate. The brain is not a receiver. The brain is a prediction engine. What the senses contribute is the corrective feedback that keeps the predictions tethered to what is actually out there.

Clark goes further. The experience of the world, he argues, is controlled hallucination — the brain's own model, generated continuously, kept in check by the sensory data that arrives to confirm or refute it. The phrasing is provocative on purpose. What the brain is doing all the time, in ordinary perception, is what happens in hallucination when the checks fail. The mechanism is not different. The difference is whether the corrections are working.

The strict three-level hierarchy described above — surface, middle, deep — is a simplification. The cortex has six anatomical layers, and the flow of predictions and prediction errors runs across them in more complex ways than the simplified picture suggests. The principle holds — predictions descend, errors ascend, hierarchies of timescale exist — but the diagram is approximate. The technical literature on cortical microcircuits is intricate and still being worked out.
· · ·

The brain does this. But it is worth pausing to ask why a brain would do this in the first place. Why would evolution build an organ that generates the world rather than just receiving it? Why not the obvious design — eyes that see, ears that hear, a brain that processes whatever the senses deliver?

The answer comes from a deeper principle that operates on anything that maintains itself against its environment — anything that does not simply dissolve into what surrounds it. The principle was articulated in its most ambitious form by the British neuroscientist Karl Friston, working at University College London across the 2000s. Friston was not, originally, asking about brains specifically. He was asking a more general question. What follows for any entity that persists across time?

Start with what it means to persist. Every persistent thing has a boundary. A cell has a membrane. A body has skin. A storm has an edge — the difference in air pressure that defines where the storm ends and ordinary weather begins. A country has borders. A river has banks. The boundary is what makes the thing a thing. Without it, no thing — just an undifferentiated continuation of environment. To persist is to maintain a boundary.

But environments are not gentle. Environments are constantly trying to disrupt the boundary. The cell's surroundings pull water across the membrane, push molecules in and out, shift in temperature and pH and pressure. The body's environment delivers heat, cold, food, threats, the slow grinding of gravity on the skeleton. The storm's environment pulls energy out of it through evaporation and turbulence. The boundary is not a thing that exists once and then continues; the boundary is something that has to be actively maintained, moment to moment, against forces that would dissolve it.

For the cell to maintain its boundary, it has to do the right thing in response to what the environment delivers. It has to let water in when it is dehydrated, expel it when it is bloated. It has to detect glucose and import it. It has to detect threats and respond. Every one of these responses is, implicitly, a prediction about the environment — the cell behaves as if it knows that water arriving from outside will hydrate it, that glucose arriving will fuel it, that this molecular pattern indicates a threat. The cell does not have a brain. The cell does not deliberate. But the cell's behaviour is, in its mathematical structure, indistinguishable from a system that holds a model of its environment.

This is the move Friston made. Any system that maintains a stable boundary must, as a mathematical consequence of being bounded, carry an implicit picture of what lies across the boundary. Not because it has decided to. Not because it has any capacity for representation. Because being bounded requires it. The maintenance of the boundary requires doing the right thing in response to what arrives from outside — and doing the right thing requires, implicitly, modelling what is outside.

The formal claim is one sentence. In Friston's 2010 paper in Nature Reviews Neuroscience: "Any self-organising system that is at equilibrium with its environment must minimise its free energy."

Free energy, in this technical sense, has nothing to do with the everyday meaning of the term. It is not the free energy of thermodynamics either, though the mathematics is related. In Friston's framework, free energy is essentially the gap between what a system expects and what is actually happening to it. Minimising free energy is minimising surprise. A system that fails to minimise free energy — a system that is continually shocked by what its environment delivers — cannot maintain itself. It either updates its predictions until they match better, or it acts to make the environment match its predictions, or it loses coherence and stops being a discrete thing at all.

This is not a story about brains specifically. The principle applies to anything that persists. The cell does this with biochemistry — it does not have a nervous system, but it predicts and acts and corrects, in the slow timescale of molecular response. The plant does this with growth — it predicts where light will be and grows toward it, with feedback adjusting the growth when the prediction fails. Your immune system does this — it carries an implicit model of which proteins should be inside the body and which should not, and it acts to maintain that distinction. The brain does this with such richness and depth that what runs on it became conscious. But the underlying principle is older than nervous systems. The brain is one substrate in which a much more general principle is running.

The boundary itself — the formal mathematical structure separating inside from outside — has a name in Friston's framework. It is called the Markov blanket. The Markov blanket is the set of variables that statistically mediates everything passing between the system and its environment. Everything outside the system that affects the system has to affect the blanket first. Everything inside the system that affects the environment has to do so through the blanket. The blanket is the channel through which inside and outside speak to each other — and the existence of a stable blanket is what makes the system a separate thing in the first place.

The technical term is precise but it can be set aside. What matters is the underlying observation: any system that holds a stable boundary against its environment will, as a mathematical consequence, carry an implicit model of what lies across that boundary. Not because it tries to. Because it is bounded. The brain is one such system. So is the cell, the plant, the immune system, the storm. Each of them is, in its own way and at its own scale, doing what predictive processing describes the brain doing — predicting the environment, acting to maintain itself, updating where prediction fails.

Predictive processing as a description of cortical function is well-established and supported by a substantial empirical literature. The free energy principle in its strongest form — as the universal principle governing all self-organising systems — is more ambitious and more contested. A number of philosophers and theoretical neuroscientists accept the cortical claim while declining the universal one, on the grounds that the FEP's strongest formulation may be true by mathematical definition rather than empirical discovery. What is settled: brains do predictive processing. What is more open: whether the same mathematics is the right framework for understanding cells, plants, and storms. The investigation of this question is active.
To be a thing — to persist against an environment that is constantly disrupting your boundary — is, mathematically, to carry a model of the thing trying to disrupt you.
· · ·

Sit with this for a moment. The brain you are reading this with is doing what the previous movements described. The room you are in is the brain's current best prediction. The body the brain is being read inside is the brain's current best prediction. The words on the page are the brain's current best prediction, updated rapidly as your eyes track across them and the next words turn out to be what was expected or not what was expected. None of this is being delivered to you by the senses. The senses are correcting the brain's predictions where they fail. The bulk of what you are experiencing right now, in this moment, is generated by your brain — not received from the world.

The implications of this take some absorbing. Several of them are worth naming, because each one is consequential for how the next essays in this series will proceed.

The senses do not deliver the experience. They correct it. Most of what you experience moment to moment is being generated internally. Sensory data is the editorial input to a story the brain is already telling. This means that the bandwidth of conscious experience is mostly the bandwidth of brain-generated content, not the bandwidth of sensory data. Your eyes deliver an enormous amount of information per second, but most of it is not what you experience — most of it is checking-data, used to confirm or correct the predictions that are the experience.

Perception and action are the same process. When you reach for a cup, the brain is not first perceiving the cup and then deciding to reach. The reach is itself a prediction — my hand will be at the cup — that the motor system makes true. Your movements through the world are predictions about the world that your body is making come to pass. This is why skilled action looks effortless from outside and feels almost passive from inside. The skilled tennis player is not calculating where the ball will be and then deciding where to move. The skilled player is predicting where the ball will be and the body is moving there as part of the prediction.

Your dreams are perception with the sensory correction turned off. The predictive machinery keeps running through sleep. What you experience in dreams is the brain's model, allowed to run free, with no incoming sensory data to correct it. The model generates rooms and people and conversations and emotional weight — the same machinery that generates them in waking life, no longer tethered to what the eyes and ears are actually delivering. This is why dreams feel real while you are in them. The mechanism producing the experience is the same mechanism that produces waking experience. The difference is whether the senses are correcting.

Hallucination is the same machinery, weighted differently. The difference between perceiving and hallucinating is not the mechanism — it is whether sensory data is being given enough weight to override top-down predictions. Strong predictions, weak sensory input — or strong predictions that the sensory input simply cannot disconfirm — and the brain generates experiences that have no sensory cause. In the case of psychotic hallucination, this can be deeply distressing. In the case of religious or mystical experience, the same mechanism may be producing experiences that have profound personal meaning. The mechanism is not the meaning. The mechanism is the machinery the meaning rides on.

And belief shapes perception. Not metaphorically — mechanically. What you expect to see affects what you see, because the brain uses expectation as its starting point and only corrects where it fails. Strong beliefs are highly weighted predictions. They resist disconfirmation by sensory data, because the data has to be strong enough to overcome the prediction's weight. This is why people in the same room can experience different rooms. This is why a frightened person and a calm person walking down the same street are not having the same walk. This is why ideology shapes perception in ways that the people inside the ideology cannot easily see.

You have never met the world. You have only met your brain's best guess at the world, updated where the senses found it wrong, running as your experience everywhere else.

There is a further question this raises, and it is the question the next essay turns on. The brain has been described, in this essay, as a hierarchical prediction machine running models at every level — the position of the cup, the shape of the room, the body sitting in the chair, the situation of being a person reading these words. Every one of these is a prediction. But the prediction of being a person reading these words is special in a way the others are not. It is not a prediction about something the senses can report on. It is a prediction at the foundation — the prediction that everything else is referenced to.

What kind of prediction sits at the deepest layer of the hierarchy, beneath the prediction of the room and beneath the prediction of the body, holding everything else in place? What is the prediction of the self?

Recode Reality  ·  Ten Thousand Things Inference Machine  ·  Complete चैतन्यम् आत्मा Caitanyam ātmā