The mathematical principles of information

For the first decade or so of my working life I was a cryptographer, and the thing I was paid to produce was randomness. It is a strange trade. Most engineers spend their careers taking noise out of a system. A cryptographer spends his putting it in, and the quality of his work is judged by how little anyone can learn from it. The pseudo-random number generator I built in those years was used far more widely than I expected, and the test it had to pass was brutal in its simplicity. Take its output, hand it to the cleverest adversary you can imagine, give them as much of it as they like, and if they can find any pattern at all, any regularity that lets them predict the next bit better than a coin toss, you have failed. Good randomness is information that will not yield to study. That definition has been in my head for 30 years, and it turned out to hold the key to a problem I did not meet until much later.

The problem arrived through aircraft. After cryptography I spent the better part of two decades building CoreAVI, a company that made safety-critical software for avionics, the graphics behind the displays a pilot flies by. In that world a system is not judged on how clever it is but on whether it can be shown, to a regulator, never to do the thing it must not do. When artificial intelligence became useful enough to be tempting, my colleague Ken Wenger and I tried to put it to work on the certification problems we understood best, and we found that the more precision we demanded of it, the more erratic it became. Every change to a prompt, a dataset or a layer of governance carried consequences we had not intended. The systems were not broken. They were doing exactly what their mathematics required, and the mathematics was the problem. What follows sets that mathematics out in plain terms. The principles that make modern AI work and the principles that bound its reliability are the same principles read in opposite directions.

What the Wire Does Not Need to Know

Every account of information begins with Claude Shannon, and it should. In 1948, at Bell Labs, Shannon published A Mathematical Theory of Communication, and everything you own with a battery in it descends from that paper. He was solving a concrete problem, how to send a message down a wire or through the air as efficiently and reliably as possible in the presence of noise, and to solve it he needed to measure how much a message tells its receiver. His answer was built on uncertainty. A message you could have predicted in advance carries nothing. A message drawn from a vast field of equally likely possibilities carries a great deal. He called the measure entropy, borrowing the word from thermodynamics, and on it he built compression, error correction and the whole idea of the bit.

To make the theory work, Shannon made a decision that he stated openly and that the rest of us have since forgotten he made. He set meaning aside. From the telephone company’s point of view it makes no difference whether the sentence on the line is a marriage proposal or a stock quote. What matters is that it arrives as it was sent. The wire does not need to know what the words mean. That was the right decision for the problem in front of him, and I have no quarrel with it. The trouble came later, when a theory built to carry messages was adopted, with little examination, as a theory of what a machine does with a message once it has arrived. Shannon told us how to move information without caring what it meant. He never claimed to tell us how a receiver comes to understand it, and the field that built modern AI has proceeded, for the most part, as though he had.

A cryptographer has a particular reason to notice this. Entropy was my raw material, and I knew from the inside that Shannon’s number measures how much a signal varies and says nothing about why. My generator’s output and the static on an old radio have the same entropy. One of them is the most carefully engineered artefact I ever produced and the other is nothing at all, and the number cannot tell them apart, because it was never meant to. Once you start looking for what else that number cannot tell apart, you find the whole subject of this essay.

One Word, Three Things

Look at information from the receiver’s side, from the point of view of a system that must act on it rather than carry it, and it stops being one uniform substance. It comes apart into three kinds, complexity, entropy and ambiguity, which I have set out in the primer The Three Faces of Information, and one sentence on each will serve here. Complexity is the learnable part. Entropy is randomness in the cryptographer’s sense, information that will not yield to study, and it cannot be optimised away. Ambiguity is when the world has already decided and has not given you enough to tell which decision it made; in the mathematics it is the uncertainty about the cause given the observation, a different quantity from Shannon’s, which is the uncertainty about the observation itself. Entropy and ambiguity demand opposite responses, and a confidence score cannot tell them apart. It was never built to.

One Number to Rule Them

Underneath its varying surfaces, almost all of modern AI is doing one thing, and that thing is optimisation. A system is given an enormous number of examples, a single number that measures how wrong it currently is, and a procedure, gradient descent, for adjusting billions of internal weights so that the number goes down. An image classifier is scored on how often it labels the picture correctly. A language model is scored on how well it predicts the next fragment of text. The engine is the same in both cases. Define one number that measures failure and grind it down as far as it will go.

This deserves its due, because it is one of the great intellectual achievements of our age. Tasks that were held up for decades as the frontier of machine intelligence, recognising a face, translating a language, holding a coherent conversation, have fallen one after another to this single idea applied at ever greater scale. Feature extraction, the compression of raw input into layers of abstraction, is what the optimiser learns to do on its way to a lower score. Statistical prediction of the next word is what it does when the score is defined over text. None of these is a separate principle. They are what a scalar objective, pursued relentlessly, produces. The success is real, and any account that sneers at it should not be trusted. An enormous amount of the useful world is complexity in disguise, patterns dense enough to look like thought, and on that ground the machines are not merely adequate but superhuman.

Now notice what the scoring leaves out. The single number rewards the system for being right more often and punishes it for being wrong, and it says nothing about why the system was wrong on the occasions it was, or whether a given mistake was the natural result of a situation that admitted no right answer. To the optimiser, the world is complexity all the way down, structure to be learned, error to be reduced. Entropy and ambiguity do not go away because the optimiser ignores them. They sit in the error term, unexamined, and the machine walks straight into them with the same confidence it wears everywhere else.

Ken and I saw this before we had a theory for it. In the early days we were simply trying to build vision systems that made fewer mistakes, and a research assistant of ours, Katy, ran hundreds of experiments on a standard image classifier, the kind that reads handwritten digits. She was thorough. Alongside the tidy test set she fed it squares of pure random noise, and images she had deliberately blurred, and she logged the pairs the model reliably confused, the way a hastily written 1 and a 7 confuse each other. Her raw results sat on a hard drive for three years before I went back to them, and when I did they shocked me, because it meant we had had the data long before we had the theory. The confusable pairs were ambiguity, figures underdetermined even for a person. Show a steep 1 to two people and you may get two answers, which tells you the trouble is in the figure and not the reader. The noise was entropy. The model, shown squares of random pixels, images that were nothing at all, announced with high confidence that it was looking at a digit.

Three kinds of information went in. One voice came out. That is the failure in a sentence, and its two halves deserve names because they are the two things a scalar objective cannot help doing. The first we call compounded entropy. Noise, which carries no learnable pattern, is mapped by a pattern-finder onto a confident answer, and randomness is made solid. The second is compression-amplified ambiguity. An input that supports two readings is forced, by a system permitted only one output, to throw away the fork that was the true state of affairs, and a real plurality is flattened into a false certainty. A large language model is the same engine at colossal scale, and when it invents a citation that does not exist and presents it with the same even confidence as a real one, it is not a different kind of machine having a bad day. It is the digit made of noise, told in sentences. The industry’s word for this, hallucination, makes it sound like a curable ailment.

Accountancy, Not Alchemy

At this point the reasonable reader pushes back, and the pushback deserves its strongest form, because it is the most successful prediction in the recent history of the field. Again and again people have pointed at something AI could not do and called it a wall, and again and again the wall has fallen to more scale. The careful version of the argument is not that scale is magic. It is that most residual failures are failures of insufficient learning, that the human brain is itself a very large network trained on a lifetime of data, and that the gap between machine and mind is therefore a gap of degree rather than of kind. On that view the confident digit made of noise is a temporary embarrassment, the sort of thing a bigger model with better training will learn not to do, as it has learned not to do a thousand other things that once looked fundamental. I held a version of this view myself for longer than I would now like to admit, and I did not abandon it because of a feeling about machines and minds. I abandoned it because of a piece of mathematics old enough and solid enough that nobody seriously disputes it.

It is called the Data Processing Inequality, and its meaning can be carried in one sentence. No amount of processing can add information about something that the processing has already thrown away. Picture the chain of events inside any of these systems. There is a cause in the world, the digit somebody wrote, the object in front of the camera. The cause produces an observation, the pattern of light that lands on the sensor. The machine turns the observation into an internal representation, a compressed summary in its own private code, and reads its answer off that. Cause, then observation, then representation. It is a one-way street. At each step some information is lost and none is added, because a photograph is not the thing it depicts and a summary is a form of forgetting. What the inequality guarantees is that the leakage cannot be reversed by any cleverness downstream. Once the summary has discarded the fact that distinguished a 1 from a 7, no further computation can recover it, any more than you can reconstruct a shredded document by studying the confetti more intently.

This is why the ambiguity failure is not a bug awaiting a patch. If the evidence was consistent with both a 1 and a 7, no architecture, no scale and no training regime can extract a certainty that was never present in the evidence. That part of the result is unconditional. The related claim about noise, that compression manufactures false confidence out of randomness, needs one further condition, but it is a condition satisfied by essentially every real network, because every real network squeezes a high-dimensional input through a much narrower internal code. The squeezing is not incidental to how these systems work. It is how they work. Compression is where the power comes from, and it is also, provably, where information about the true cause is shed.

People sometimes hear this as a mystical claim, as though I were saying the machine can never know anything, and that is not it. The inequality is not romantic. It is accountancy. It tracks a quantity, the amount one thing tells you about another, and observes that the quantity can only fall as you move down a chain of processing. If the pixels contain enough to pin down the digit nine times in ten and no more, nothing computed from those pixels can do better than nine in ten. Computation is not alchemy.

Put the pieces together and you have a ceiling. Within the whole family of systems built around a single objective and a single best guess, accuracy does not improve without limit as you add data and scale. The ceiling is set not by the budget or the cleverness of the engineers but by the irreducible entropy and ambiguity in the problem itself. On a task that is nearly all complexity the ceiling is so high you may never bump into it, and the machine will look, for every practical purpose, perfect. On a task laced with real randomness or real underdetermination the ceiling is lower, and none of the usual medicine lifts it. That a ceiling exists is a proved consequence of the mathematics, given assumptions that hold in the machines being built and sold today. Where it sits for any particular task, a medical scan, an open road, a certification document, is a live research question, because measuring the irreducible uncertainty in a real problem is itself hard.

I should say where this sits against the field’s own work, because none of it arrives in an empty room. Machine learning has long separated aleatoric uncertainty, the randomness in the data, from epistemic uncertainty, the model’s ignorance; Alex Kendall and Yarin Gal set the distinction out for deep networks in 2017, and it is the right place to start. Around it sits a toolkit for letting a model decline: out-of-distribution detection, which flags inputs unlike the training set; conformal prediction, which returns a set of answers with a guaranteed coverage instead of one answer with a confidence; abstaining classifiers and the learning-to-defer literature, which hand hard cases to a person. All of it exists and much of it works. The ceiling described above is not a claim that the field never built an “I don’t know”. It is a claim about what no scalar can carry: the epistemic reading and the ambiguous reading of the same input produce the same number, and the difference between them is not in the model’s ignorance, since ambiguity survives a model that predicts every appearance perfectly, but in the geometry of the problem. The out-of-distribution detectors are the nearest neighbour to what the second system does, and the difference is worth being exact about. They catch one way that entropy can arrive, the input unlike anything seen before. The entropy that costs most sits inside the distribution or close to it, the steep 1 that is also a 7, where a novelty detector sees nothing at all, and what the geometry reading supplies is the signal in that case: not only that uncertainty is present but which kind it is, and therefore what to do.

The Arithmetic of Long Answers

There is a second piece of arithmetic, simpler than the first, and it explains why the failures grow worse as the systems are asked to do more. A language model does not produce an answer. It produces a sequence of commitments, one token after another, each conditioned on the last. Suppose each token has a 98 per cent chance of being acceptable for the meaning intended. That sounds excellent. Multiply it across a 200-token answer and the probability that the whole sequence is acceptable is a little under two per cent. Raise the per-token figure to 99 per cent, which is a large improvement and roughly what a good fine-tuning pass buys you, and the 200-token figure rises to about 13 per cent. Better, and still not odds on which anyone would sign a certification. Precision compounds multiplicatively across length. Fine-tuning, prompting and reinforcement learning shift the per-token number upward, sometimes a long way, but they do not change the shape of the curve, which decays exponentially with length and with the residual complexity, entropy and ambiguity that no training removes.

This is why a system can score brilliantly on a test of single questions and then fall apart when asked to carry out 200 consecutive decisions. It is the arithmetic behind every agent that looks promising in a demonstration and comes undone in production. At each step there is a small probability of slipping out of alignment with the intent, and the steps do not average. They multiply. Ken and I proved, under assumptions that describe the machines in use today, that no system governed purely by a scalar objective can hold its alignment over a long enough horizon, and when we built a benchmark to measure it, testing how many steps of a fully deterministic task a model could sustain, the decisive variable was not size. Across 29 models from eight providers, one large frontier model with hundreds of billions of parameters collapsed after about six steps. A much smaller model built to hold and check its own reasoning sustained well over 100. Under the strictest scoring, every model eventually drifted. That is not a defect awaiting a patch. It is what a commitment accumulator does when nothing inside it is keeping watch.

Teaching a Machine to Say No

All of which sharpens the question rather than answering it. If the mathematics forbids a single-objective system from telling noise from ambiguity, what is to be done? The answer we arrived at surprised me, because I expected it to lie in a better optimiser and it lay in reading the one we already had more carefully.

At Squint Cognition, the company Ken and I built around this work, we have a tool that lets you look inside a trained image model and watch how it organises what it sees, not as a label and a confidence but as geometry, the layout of the model’s internal states as points in a space, near for things the model finds similar and far for things it finds different. One of our small test classifiers had been trained to tell M&Ms from Skittles, and it was good at it. Then somebody showed it coins. At the level of its single forced answer the model did exactly what the mathematics predicts. It sorted each coin into one sweet or the other, untroubled, wrong and sure of itself. In the geometry the coins did something the answer never revealed. They did not scatter, and they did not land on top of the sweets. They gathered, off to one side, into a small coherent cluster of their own. The model had, in a real sense, noticed that it was looking at something new. The noticing was simply thrown away at the last step, when everything it knew was compressed into one label.

That is the seed of the practical answer, and its logic follows directly from the Data Processing Inequality. The loss of information inside a network is gradual, and the layers just before the final collapse still hold a great deal that the last step discards. Read the model one step before it commits, where the distinction between a clean case and a doubtful one is still legible, and you are not adding information the model never had. You are catching it before it destroys the information it did have. Ken tested this on the plainest cases first. Run the classifier as usual and, alongside it, read where the input falls in the geometry. Most of the time the two agree. Where they disagree, where the forced label says one thing and the geometry says the point is out at a boundary or between two clusters, do not force a single answer. Flag the case and hand on a short candidate set for something else to resolve. On the handwritten digits, for the troublesome numeral eight, that rule cut the errors from 16 to four with no change to the underlying model. In a study we published, the same approach wrapped around an ordinary object detector on a public crop-and-weed dataset lifted accuracy on the cases the detector found hardest from about 67 per cent to about 78 per cent, and the more valuable half of the result was not the lift but the sorting. The system knew which cases were hard, which is the one thing a bare confidence score never tells you. These are published experiments, built and evaluated by us and not yet independently replicated, and I would rather report them at that grade than a grander one.

The general shape is a second system. An optimiser cannot ask whether optimising is still the right thing to be doing, for the same reason a ruler cannot measure its own length. The question requires a vantage point the optimiser does not contain. What has to be added is a layer that holds a standing representation of the intent, evaluates candidate outputs against it by some standard other than the training objective, watches for drift as the steps accumulate, and has the power to act on what it sees. Acting means the system gains more than one thing to say. It can commit, when the input sits inside what it knows and the geometry agrees. It can register, when something coherent but unfamiliar appears, marking it as new rather than jamming it into an old box. It can signal, when noise or ambiguity has overwhelmed its ability to place the input, and the right output is not an answer but a flag. Commit, register and signal are three responses where there was one, and they map onto the three faces of information. Every hallucination is a commit that should have been a signal. Every confidently mislabelled coin is a commit that should have been a register.

Take a domain where the stakes are plain. A model reading chest X-rays meets a scan whose features sit in a region where its own geometry says the evidence forks. The single-voice system picks the likeliest reading and reports it with a number. The governed system reports that this scan is one of the ones on which it should not be trusted, and sends it to a radiologist with the two or three readings it could not choose between. The second system is not cleverer than the first. It is differently placed, an auditor looking at the accountant’s books, asking a different question. Is this trustworthy? Is this within scope? Should we act on this yet?

Uncertainty, on this view, stops being a defect to suppress and becomes a signal to read, and the signal is not one thing. Unexpected uncertainty can mean danger, where the situation has moved outside anything the system can handle and the right response is restraint. It can mean correctable drift, where the world has shifted a little and the system can adjust without tearing anything up. It can mean opportunity, a persistent structured surprise that says there is something real here that the current picture does not contain. A system that collapses these three into one, treating every surprise as an alarm or every surprise as a chance to adapt, is either paralysed or reckless. The skill is in the distinction. Stated that way, the governance of uncertainty starts to look less like a failure of cognition and more like its substance.

The second system is not consciousness, and it does not pretend to be. A machine that can say it does not know is not thereby a machine that knows what it is like not to know. What the layer supplies is the minimal, functional, buildable core of something we usually bundle up with grander words, the capacity of a system to evaluate its own epistemic state and act on the evaluation. That is a smaller achievement than the field’s louder voices promise and a more important one than its overlooked failures suggest.

Which Kind of Not Knowing

The cryptographer’s test, the one my generator had to pass, was that no adversary could learn anything from its output. I spent a decade making information that would not yield to study, and I have spent the last several years watching machines that cannot recognise such information when they meet it, machines built so brilliantly to always see something that they were never once built to see nothing. The two halves of my working life turn out to be the same subject. Shannon gave us a number that measures how much a signal varies and, by his own design, nothing about why. The mathematics that descends from him, the scalar objective and the compressed representation, is a magnificent engine for the one face of information that yields to study. It is blind, provably and not temporarily, to the two faces that do not.

None of this diminishes what has been built. Machines now handle complexity better than any person, and that is a real achievement no amount of sceptical mathematics takes away. The industry has also mistaken competence at one face for competence at three, and staked a great deal, in cars and clinics and courtrooms, on a confidence the machines are not equipped to earn. The repair is not more of what created the problem. It is a second system whose whole job is to notice when the machine has left the ground it understands and to say so.

There is a line of Ken’s that has become the motto for this work. Fluency is not awareness. A system can grow more articulate without limit and none of it adds up to the one capability that matters for trusting it, the ability to know when it should stop. That capability does not arrive with scale, because it is a different kind of thing from the thing scale improves. It has to be designed in.

You can test the three faces without any equipment. The next time you are somewhere familiar, notice how much of what reaches you costs you nothing, because you have learned its patterns. Then notice the moment when something will not resolve, when no amount of staring tells you what happens next, and watch what you do with it. Then notice the moment when the same evidence would fit two stories and you have to choose or wait, and watch what you do with that. Three kinds of not-knowing, three responses, running under everything you think. A mind is a thing that learned, over a very long time, to tell them apart. Our machines have so far learned to handle one, and the question worth asking of any of them, before the question of how accurate it is, is which kind of not-knowing it is in.