The evolution of complexity, entropy, and ambiguity in a perception system

On 7 May 2016, on a divided highway near Williston, Florida, a car driving itself ran at full speed into the side of a white articulated lorry that was crossing its path. The trailer was tall and pale, the sky behind it was bright, and to the car’s vision system the trailer was the sky. It did not brake. The driver was killed. Ken Wenger and I began pulling at the thread that became Cognitive Information Theory not through philosophy but through wreckage like this, because self-driving cars are the purest test yet built of a machine’s ability to meet the open world, and their failures were not scattered at random. They gathered at boundaries. Something about the information was giving way at particular places, and I wanted to know where.

The lorry was there. Its whiteness was real. The light that reached the camera was real. Somewhere between the trailer on the road and the decision not to brake, the fact that mattered, a solid object ahead, went missing, and the question this essay asks is where along that road it was lost. There are three places it could have gone. It could have been absent from the world as the sensor met it. It could have been discarded by the sensor. It could have been thrown away by the model. Each of those is a different failure, each calls for a different repair, and a system that cannot tell which one it has suffered will be repaired in the wrong place. What follows walks the road that information travels, from the world to the instrument to the mathematics, and watches what happens to its three faces at each step.

Three Places Information Can Be Lost

Ken and I describe information by its three faces, complexity, entropy and ambiguity, which I have set out in the primer The Three Faces of Information. Every situation a perception system meets is some mixture of the three, and the mixture is not fixed. It changes as the information moves.

It moves through three states, and this is the part of the theory that turns a vague sense of loss into something you can point at. There is information as it exists before anyone looks at it, which I will call unobserved. There is information as a particular instrument has captured it, which is observed, and which already carries the perspective and the limits of that instrument. There is information as a model has interpreted it, mapped onto categories, labels and decisions, which is interpreted. A self-driving car, a factory inspection camera, a radiologist’s workstation and the human visual system all have the same three-stage shape. The world, then the instrument, then the interpretation. At each of the two transitions the three faces are transformed, and never in the direction one would hope. Complexity is reduced, which is the point. Entropy is not removed, only hidden. Ambiguity is either preserved or, more often, collapsed into a false certainty. By the time the lorry reached the decision, it had passed through both transitions, and the essay’s claim is that it was lost at the second and made unrecoverable at the third.

The World Before Anyone Looks

Begin where the information begins, in the world itself, before any sensor is pointed at it. The real world is not complicated in the way a spreadsheet is complicated. It contains an unbounded number of details at every scale, from the precise angle of the sun to the micro-texture of the road surface, and everything in it is related to everything else. Wind moves a tree, the tree casts a shifting shadow, the shadow falls across a lane marking, and the lane marking now looks different from the one in the training data. Complexity at this level is relational, and it is effectively infinite. No instrument captures all of it and no model could hold it if one did.

The world is also not deterministic from where any observer sits. The same street is a different street in rain, at dusk, with a school bus stopped on it. A pedestrian steps out. A tyre fails three cars ahead. These are entropy in the sense the theory uses the word, the part of a situation that no amount of driving the same road makes predictable, because the world is producing it without a pattern for the observer to catch. A perception system built to find patterns will keep looking for one here, and it will find one, because finding patterns is what it does.

The world is also ambiguous before anyone looks at it, and this is the face people most often mistake for a sensor problem. A black plastic bag lying flat on the road and a pothole present the same dark shape to the road ahead. A white trailer against a white sky and a stretch of open air present the same brightness. The ambiguity is not created by the camera. It is in the geometry of the scene, in the fact that two different states of the world produce one appearance from that angle in that light. Humans resolve this kind of ambiguity constantly, using context, intent and the ability to wait for one more piece of evidence, and they resolve it so automatically that they never notice a choice was made. Raw information encodes none of that. It carries the appearance and leaves the choosing to whoever arrives next.

Put those three together and you have a way of saying something the industry has never had a good way of saying, which is how hard a task is. Consider driving a car through a busy intersection, driving the same car along a motorway, and driving across an empty desert, which was the original challenge the American defence research agency set for autonomous vehicles in 2004. Anyone can see that these differ in difficulty, and nobody can say by how much, because there is no accepted measure of the magnitude of a real-world task. What the field does instead is quote behaviour. A model is 90 per cent accurate. A car has driven 1,000 hours without incident. Neither statement says anything about the world the system was measured against, and 1,000 hours on the motorway and 1,000 hours in the city are not the same thousand hours. The three faces give a better first question. How much of this environment is complexity, which the system can learn, how much is entropy, which it cannot, and how much is ambiguity, which it will be forced to collapse?

Ask that question of a factory floor and the answer is striking. A camera watching parts pass under a fixed lamp on a conveyor faces almost none of the world’s variety. The lighting is constant. The motion is controlled. The set of things that can appear in the frame has been decided in advance by someone who owns the building. Complexity is high but bounded, entropy is low, and ambiguity has been engineered out, because a part either meets its specification or does not, and the two cases were chosen to look different. This is why machine vision on a production line has been reliable for decades while machine vision on a public road is not. The models are not smarter. The world in front of them has been made simpler, and the three faces measured at the source are small.

The Instrument Chooses

The second stage is the one I came to think most underrated, and the theory treats it as a first-class component of intelligence rather than as plumbing. Between the world and the model sits an instrument, a camera, a laser rangefinder, a radar, an eye, and the instrument does not deliver the world. It delivers a selection. Every act of observation chooses what can be seen and, in choosing, excludes what cannot. A camera captures a range of light, at a resolution, at a frame rate, from a position, and everything outside those choices is gone before any mathematics is applied. What the instrument hands on is observed information, and observed information is smaller than the world in ways the observer cannot see, because the parts it cannot see are exactly the parts that were removed.

Consider the same field of tall grass seen by a person and by a hawk. The hawk, with far better eyes, resolves a mouse on a stalk that is, for me, lost in an undifferentiated blur. What is complexity to the hawk, learnable structure it can act on, is closer to noise for me. Information is not simply out there. It is relative to the instrument that meets it, and the face it wears depends on who is looking. A 1080p camera sees far less than a human eye. A laser rangefinder returns the outline of an object and nothing of its colour or texture. Radar returns range and speed and is confused by reflections. Each instrument turns the same world into a different observed set, and each observed set has its own mixture of the three faces.

The transition does three things at once. It reduces complexity, which is necessary and useful. The camera does not record individual raindrops, and most of the time nobody wants it to. In heavy rain, though, the detail it discarded was the detail that separated the lane marking from the wet road, and the reduction that was harmless in sunshine is now the reason the lane is lost. It adds entropy of its own. Cameras blur under motion, laser returns are sparse at distance, radar reflects from the wrong surfaces. Some of this can be averaged away by looking over several frames, at the cost of reaction time, and that trade between noise and delay is a real one that no design escapes. Above all, it alters ambiguity, and it can alter it in either direction. A low-resolution image turns a pedestrian and a pole into the same blurred upright, which is ambiguity the world did not contain, manufactured by the instrument. Two instruments together can do the opposite. A camera alone mistakes a reflection for an object. A rangefinder does not care what colour a trailer is; it reports a solid surface at a measured distance. Fuse the two and much of the ambiguity that either alone would have carried forward is resolved at the interface, before the model ever sees it, because the two instruments lose different things.

Ken wrote up some notes after one of our conversations about this stage, and one image in them has stayed with me. He described the passage of information through the perception interface as a collapse of dimensions. The world reaches the instrument as a point in an enormous space, and the instrument projects it into a smaller one, the way a sphere casts a circle. The projection is lossy, and once it has been made, the higher-dimensional object cannot be recovered from the shadow unless something else supplies what was lost. Ken’s second observation was the one that mattered. A single system trained on a single objective can be very good at the forward direction, the compression from world to decision, and it cannot at the same time be good at the reverse, the reconstruction from decision back towards the world. Those are different tasks, and they pull a single model in opposite directions. Any system that hopes to notice what its instrument threw away needs, at minimum, a second part whose job is to look back.

The lorry belongs at this stage, and at the interface with the next. Against that sky, in that light, from that angle, the camera’s observed set contained a bright region where the world contained a wall. The information distinguishing the two was present in the world and absent from the pixels, or nearly so, and no downstream cleverness can restore what the instrument did not capture. The radar’s observed set was different. By Tesla’s own account afterwards, the radar returned something at the height of the trailer, and the software discounted it as an overhead sign, because a road is full of bridges and gantries and a car that brakes for every one of them is its own hazard. The fact was present in one instrument, ambiguous in the other, and the rule where the two met had been tuned to resolve that kind of disagreement in the camera’s favour. That is not a failure of the model in the usual sense. It is one instrument’s selection, made faithfully according to its physics, delivering a set in which the most important fact on the road had become ambiguous with open air, and a rule at the interface throwing away the one instrument that had not been fooled.

What the Model Does with What Is Left

The third stage is the one that gets all the attention, and it is where the mathematics of The Mathematical Principles of Information does its work. A model takes the observed set and interprets it, and the model in almost every commercial system is an optimiser of a single number. It has been shown an enormous number of examples and adjusted, billions of times, so that its errors on those examples fall. What it learns to do on the way is to compress. A network trained on road images discovers which features of a road are essential to the decisions it is scored on, and it discards the rest, folding thousands of pixels into a compact internal code from which it reads its answer. This is where the power comes from. It is also a further reduction of complexity, layered on top of the instrument’s, and the two reductions compound. Every edge case, the unusual vehicle shape, the carnival float, the trailer that looks like sky, is a place where a detail the model learned to discard was the detail that mattered.

Entropy passes through this stage untouched, because the optimiser has no way of representing it. A model trained in clear weather has never seen snow. Shown snow, it does not report that it has left the ground it understands. It looks for the nearest pattern it knows and reports that, with the same confidence it wears everywhere else, because it was built to always produce an answer and never to produce the answer that says this is nothing I recognise. The industry has begun to bolt on probabilistic machinery, networks that output a spread rather than a point, and that helps at the margin. It does not change the structural fact that a single number for confidence cannot distinguish the entropic case from the merely unfamiliar one, because the information that would distinguish them was never part of the objective.

Ambiguity is where the third stage does its worst, and the mechanism is simple. The model is permitted one output. An input that the observed set left consistent with two readings, pedestrian or pole, obstacle or sky, has to be mapped onto one of them, and the mapping throws away the fork that was the true state of affairs. We call this compression-amplified ambiguity, and the word amplified is deliberate. The world contained a fork. The instrument may have widened it. The model erases it, and reports the erasure as certainty. In a factory this scarcely matters, because the input was engineered so that the fork is rare and the decision is pass or fail. On a road in fog, a vaguely human-shaped object ahead is exactly a fork, stop or continue, and the model’s forced choice is the whole of the outcome. There is a piece of mathematics, the Data Processing Inequality, which belongs to Shannon’s information theory and is the oldest thing in this essay, that says this cannot be fixed by a better model. No processing can add information about something the processing chain has already discarded. If the pixels were consistent with both a trailer and a sky, nothing computed from the pixels can extract a certainty that was never in them. The best a model can do with a real fork is to notice it is a fork, and a single-objective model is not built to notice.

The neighbours here are the sensor-fusion literature, which has understood for decades that instruments losing different things can be combined, and the uncertainty toolkit of machine learning, which separates the randomness in the data from the model’s own ignorance and has built detectors for inputs unlike anything the model was trained on. Both are real and both are used on real cars. What the three faces add is the placement of the losses at the transitions, world to sensor to representation to decision, and a reason the novelty detectors are not enough: the trailer against the sky is not out of distribution. It is a perfectly ordinary input that happens to sit on a fork, and a detector built to notice the unfamiliar will pass it through. The entropy that kills is usually in distribution, or close to it, and the reading a second system takes one step before the model commits is a reading of that case, not of the strange one.

Why the Factory Is Easy

Lay the two systems side by side along the three stages and the difference between them stops being mysterious. The factory camera starts from a world in which the three faces have been made small by design, sees it through an instrument that loses almost nothing that matters, and is asked a question whose two answers were chosen in advance to look different. The car starts from a world whose complexity is unbounded, whose entropy is high and whose ambiguity is intrinsic, observes it through an instrument that manufactures new ambiguity while removing old detail, and collapses whatever survives into a single confident action, at speed, with a person inside. This is why I have come to think the argument about self-driving cars is being conducted at the wrong stage. Almost all of the effort is spent on the model, on the assumption that a better model will close the gap. Some of it will. Most of the gap is at the first two stages, in what the open road contains before any camera is pointed at it and in how much of what matters the camera then leaves behind, and a perfect model of an observed set that no longer contains the lorry will not brake for the lorry. The factory solved its problem by controlling the world and choosing an instrument that lost nothing it needed.

Two Bets on the Road

The account has two deployed tests, and it is fair to say how each bears on it. Waymo runs cars with nobody in the driving seat in Phoenix, San Francisco, Los Angeles and a growing list of cities, and it does so by doing, as far as a public road allows, what the factory did. The cars run inside a mapped and bounded territory, on roads surveyed to the kerb before the first passenger, with lidar, radar and cameras fused, and the territory grows only as the map does. That is controlling the world and choosing an instrument, and it works, which corrects a sentence I would otherwise have written: someone building a car does have that option, provided they will give up the open road for a fenced one and pay for the instruments that lose the least. Waymo’s record inside the fence is, on its own published figures and on the independent analyses so far, better than the human baseline, and I take that as evidence for the account rather than against it. The first two stages were made small, and the third became tractable.

Tesla made the opposite bet, and it is the explicit rival to the fusion argument above. In 2021 it removed radar from its cars, and in 2022 the ultrasonic sensors, on the reasoning that people drive with two eyes and a brain, that a camera-only system trained on enough miles will do the same, and that when two instruments disagree the disagreement is not information but a liability, since the system must then decide which to believe. I understand the reasoning and think it is wrong in a specific way. Two eyes are not a single instrument. The visual cortex does the fusion, and the disagreements it resolves, between the two eyes and between vision and the balance organs, are exactly what depth and motion are made of. A camera-only system does not escape the fusion problem; it moves the fusion inside the model, into the third stage, where the account says a fork is most likely to be erased. The bet may still pay, because the model may learn to hold forks that a fused system resolves mechanically, and the miles Tesla has collected are the largest observed set anyone has. What the bet cannot do is recover in the third stage what a rangefinder would have supplied in the second. Williston is the case in point. The radar was on the car in 2016 and returned something; the interface discarded it. Removing the radar does not fix that decision. It makes it permanent.

What Would Have to Change

If the losses happen at the transitions, the repairs have to happen there too, and three follow from the account above. The first is at the instrument. Two sensors that lose different things resolve ambiguity that neither alone can, which is the plain argument for fusion, and it is a stronger argument than the usual one about redundancy. The point is not that two cameras are safer than one. It is that a camera and a rangefinder disagree about a white trailer, and the disagreement is information. A system that treats the disagreement as noise to be averaged has thrown away the one signal that would have saved it.

The second is at the model, and it is not a better optimiser. Everything Ken and I have found says the missing capability is a second system, differently placed, whose job is to read the first one before it commits and to ask whether committing is warranted. A trained model holds more in its internal geometry than its final label reveals, and the layers just before the collapse still show whether an input sits inside familiar ground, out at a boundary, or between two clusters. Read that geometry and the system gains more than one thing to say. It can commit when the ground is familiar. It can register that something coherent but new has appeared. It can signal that noise or ambiguity has overwhelmed it, and hand the decision on. On a road that last option is a brake, a slowing, a request that the person in the seat pay attention, and it is worth more than any improvement in the accuracy of the commits.

The third is at the first stage, and it is a change in the question rather than the machinery. Before asking how accurate a perception system is, ask what kind of situation it is in. How much of this environment is learnable, how much is irreducible, how much is underdetermined, and how much of each does the instrument carry through? Those are questions you can often answer on paper before committing a year of engineering to a model that was never going to work, and they are the questions the factory answered, without naming them, decades ago.

Nothing here makes a perception system understand what it sees, and nothing here says that self-driving is impossible or names a date for it. A car that can say it does not know what is ahead is not thereby a car that knows what a lorry is. What the account supplies is smaller than understanding. It says where, along the road from the world to the decision, the information about the world went missing, and which of the three faces was lost. The lorry near Williston was present in the world, ambiguous to the camera, visible to the radar, and thrown away where the two met, and the system that killed a man had no part of itself positioned to notice any of that. Every perception system, in a factory, in a clinic or on a road, can be asked the same question. Where was the information lost, and does anything in the system know?