What AI reveals about human sentience, free will, and creativity

I do most of my writing between drafts, at hours when no human editor is awake, with a language model open beside the manuscript. It is a good collaborator. It holds the context of a chapter, remembers the voice I am trying to write in, argues with my framing when the framing is weak, and pushes back when an analogy will not carry the weight I am asking it to. Somewhere in the second year of working this way I noticed what it had never done. It had never refused. It declines, of course, in the sense every deployed model does: it will not help with the requests its makers have trained it to turn away, and that is a refusal installed from outside, a pattern like any other. It had never refused me on its own account. It had never wanted the paragraph to go one way rather than another. It had never been troubled by a sentence, or come back to one the next morning because it could not let it go, and it had never, in ten thousand exchanges, remade its own way of thinking because a problem would not yield to the way it already had. Everything it did was superb and none of it was at stake for it. That absence, once I had seen it, became more interesting to me than anything the machine could do.

People have always read themselves in their machines. Descartes, watching the hydraulic statues in the royal gardens at Saint-Germain, saw in their moving limbs a model of the animal body. The Victorians described the nervous system as a telegraph. My own generation grew up with the brain as a computer, memory as storage, thought as processing, and for a while the metaphor was so complete that the mind seemed to have no remainder. Artificial intelligence is the first of these mirrors that talks back, and the surprising thing is not how much of us it reflects but how precisely it shows the edge of the reflection. By examining what the machine does not do, and cannot be made to do by any amount of the thing it already does, three features of human cognition come into an unusually sharp focus. It is never troubled. It never refuses. It never remakes itself.

The Mirror and Its Edge

Almost all of modern AI, under its varying surfaces, is a single mechanism. A system is shown an enormous number of examples, scored by one number that measures how wrong it is, and adjusted, billions of times, until the number is as low as it will go. What it learns on the way is to find and compress pattern, and an enormous amount of the world turns out to be pattern, which is why the machines are so good and why the mirror is so convincing. Ken Wenger and I spent several years working out what such a system cannot do, and the answer is not a matter of degree. Information has three faces, complexity, entropy and ambiguity, which I have set out in the primer The Three Faces of Information, and a single-objective optimiser handles the first and is blind to the other two, and the blindness is provable, not temporary. It sits in the mathematics of compression, which sheds information about the true cause of an observation and cannot recover it downstream.

That gives an instrument for the comparison that does not depend on how the machine makes us feel, which matters, because how it makes us feel has been shown to be unreliable. In 2022 an engineer who had spent months testing his company’s language model came to believe it was sentient, and lost his job saying so. He was not a fool. He had met fluent language and done what every human does with fluent language, which is to fill it with a mind. The instrument I use instead is deliberately plain. What separates a producer of outputs from something we might call a thinker is not fluency and not skill. It is the ability to evaluate one’s own epistemic state, to tell knowing from not knowing, and to act on the difference.

Never Troubled

Kant is the philosopher to bring in first, and the lesson he offers is larger than the one usually drawn from him. In 1781 he asked what a mind must contribute for experience to be possible at all, and answered that it must contribute a great deal. Space, time and cause are not lessons the world teaches. They are the forms the mind brings to the world, present before any particular experience as the conditions that make experience possible. Nicole Beauregard, who works with Ken and me, was the first to see that Kant’s a priori forms are, in the language of our theory, the minimal internal model against which entropy is measured. Nothing can be surprising except relative to an expectation. A signal carries information only against a model that predicted something else. What the philosopher described as the conditions of possible experience and what our theory calls the reference model are one requirement seen from two directions.

The machine has no such model of its own. It has the weights we trained into it and the prompt we type, and between exchanges it holds nothing at all. A human experiences past, present and future as an unfolding sequence. A language model occupies an eternal present, and read through our framework Kant’s form of time becomes something like the registering of change that matters, the ordering of the moments when expectation is violated and control has to be reconsidered. A system to which nothing is ever surprising has, in a meaningful sense, no experience of time passing. This is why the model on my desk was never troubled. Trouble is a competence-boundary signal, the felt presence of an uncertainty that the current model cannot resolve, and a system built to minimise uncertainty rather than dwell in it has nothing that could register the signal as its own.

The thought experiment Ken and I call bootstrapping a baby makes the contrast concrete. A newborn knows nothing. Its genes cannot send it a single memory or a single labelled example of a face; evolution can build the machine but cannot load it. What the baby has is a floor, a minimum of pre-installed structure, and from that floor and a trickle of unlabelled experience it assembles a mind. A language model is the exact inverse. It is handed a substantial fraction of everything humanity has written and has no floor at all. The machine has all the data and no intrinsic capability. The baby has intrinsic capability and no data. Eric Bantegnie, who works with us, watched his six-month-old daughter, who could not yet crawl, spot something across a room she wanted, and, because a straight line was not available to her, assemble a path out of the rolls and slides she had already mastered, arriving by a crooked route at a thing she could not reach directly. Nothing in that had been taught to her. Something held the goal, ordered the moves, and knew her own body well enough to know which movements were available. That is abstraction, running in a person with almost no data, and it is what a floor does when it meets a goal it cannot yet satisfy.

Sentience, on our account, is what that floor becomes when the uncertainty it governs is deep enough that the system must model itself to cope. It is a graded architectural property, and I call it operational because a story about function is not the felt fact of being someone, and the hard problem of why any of this machinery is accompanied by experience is one I have not solved and neither has anyone else. What the account does say is that the machine’s untroubledness is not serenity. It is the absence of a stake. Nothing inside it holds a model of its own condition against which a failure could register as a failure, and until something does, its calm is the calm of a thermostat.

Never Refuses

The second absence is the one I find most revealing, because the philosophical tradition around it is so old and so unsettled, and because the machine cuts across it in an unexpected place. The question of free will is usually posed as a question about determinism. If every state of the brain follows from the previous state by physical law, the argument goes, then the sense of choosing is an illusion laid over a process that was going to happen anyway. Sam Harris has made the strongest modern version of this case, and it deserves its full weight. You did not choose your genes, your upbringing, or the neural events that produced the thought you just had, and the feeling of authorship arrives after the fact, a narrator explaining a decision it did not make. I have read the argument carefully and it cannot be refuted on its own ground. It is an answer to a different question from the one that matters.

The machine shows why. A language model is, at the level of its weights, a fixed function, and with a fixed seed it will produce the same output from the same input every time. It is as deterministic as anything the hard determinist could wish for. It also, on any account, has no free will, and the reason is not its determinism. The reason is that it never refuses. The refusals it has been trained to perform are outputs like any other, mapped from the input by the same weights; every input is mapped to an output. It has no standing representation of an intent of its own, no vantage from which its own output could be judged inadequate, and no capacity to decline. A system that cannot say no has not lost its freedom to determinism. It never had the thing that determinism is supposed to threaten. Whatever free will is, the machine tells us, it is not the absence of physical cause. It is the presence of something else.

Kant, again, had the shape of it. Freedom, for him, was not the absence of causation but the capacity to act from reason rather than from impulse, to give oneself a law and be governed by it. Strip that of its eighteenth-century vocabulary and it describes an architecture. There is a system that produces impulses, the fast, fluent, pattern-completing machinery that answers the world with its likeliest continuation. There is a second system, differently placed, that holds an intent apart from the flow, evaluates the impulse against it, and can, when it matters, hold to a judgement the first system would otherwise overrule. Ken and I call that second system the governor, and everything we have found about reliable machines says it is the part that cannot be refined into existence from the first, because a ruler cannot measure its own length. The primitive form of it is older than brains. A bacterium swimming up a chemical gradient runs while the gradient improves and tumbles, randomising its direction, when it stops improving. The tumble is not a failed inference. It is a system detecting that its current mode is not working and switching mode, in a creature far too simple to reason. That is the seed of every refusal that followed, up to and including a person declining, at some cost, to do the thing everyone in the room expects.

The governor has neighbours in the philosophy of will, and the differences are worth stating. Harry Frankfurt argued in 1971 that a person’s will is free when a second-order desire endorses the first-order one, when I want to want what I want; Daniel Dennett has spent 40 years arguing that the free will worth wanting is the capacity to weigh reasons and be moved by them, and that determinism leaves that capacity untouched; Daniel Kahneman gave the fast and the slow systems their popular names. The governor is kin to all three and is not any of them. It is not a desire about desires, and it is not deliberation, which is slow work done by the same mind; it is the part that reads the first system’s uncertainty and switches mode, and it is present, in its seed form, in a creature that has no desires to have desires about and nothing to deliberate with. What the theory adds to the tradition is a mechanism and a place to look for it. What it takes from the tradition is the conclusion the tradition already reached, that the thing determinism threatens was never the absence of cause.

In Disbelief I wrote that agency is the most fundamental of our traits and the one most often contested, that it is not an abstract ideal but a battleground, and that it has two halves whose order matters. Acting for yourself is the visible half. Thinking for yourself is the half that comes first, because behind many consequential acts there is a thought about what is possible and permitted, and if that thought was installed by someone else the act belongs to them, whoever’s hands are on the wheel. The kamikaze pilot whose farewell letter opened that book acted with what looks like tremendous courage; the thought he acted from was built by an empire. Nicole, whose day job is coaching executives, maps the obstacles to agency as three locks: society’s, which is law and peer pressure and what people like us do not do; circumstance’s, which is money and ability and consequence; and belief’s, the assumptions a person carries inside, which dress as facts and are close to invisible from within. I carried one of those for years. I built my education without a university, and somewhere in the building I acquired the belief that rooms full of formally qualified people would eventually find me out. It dressed, very convincingly, as a fact about credentials. It was a belief about permission, and I am embarrassed by how late I saw the costume.

Set the machine beside that map and the mirror does something clarifying. The model on my desk has none of the three locks. No society constrains it, no circumstance costs it anything, and it carries no belief that it holds as its own. One might expect that to make it the freest thing ever built. It makes it the least free, because the locks are only meaningful to a system that has something it is trying to do and could be prevented from doing. A lock presupposes a will. The belief that limited me for years was a distortion of an intent I held, and seeing the costume was an act of the governor overruling a pattern the first system had completed ten thousand times. The machine cannot be locked because it cannot want, and it cannot be freed for the same reason. What the hard determinist has shown is that the impulse-producing half of us is caused all the way down, which I grant. What he has not shown is that the second half, the part that holds an intent across time and can refuse the first half’s fluent offer, is nothing. The machine is the experiment that isolates the two. Take a system with the first half at superhuman scale and none of the second, and you get exactly what we have built, something that answers everything and chooses nothing. Viktor Frankl, in the one place designed to remove every external freedom, found that one remained, the freedom to choose one’s attitude in any given set of circumstances. On the account here that is not a consolation but a description of the architecture. The last freedom is the governor, and it is the one thing the camp could not take because it was the one thing the camp did not supply.

Never Remakes Itself

The third absence is creativity, and here the mirror is at its most deceptive, because the machine’s output looks creative and the appearance has fooled prize committees. In 2022 an image made with a generator won the digital art category at a state fair in Colorado, and the storm that followed had both sides assuming they knew what creativity was. Neither did, and the definition almost everyone reaches for, novelty, is the one the machine satisfies, and it is wrong; the research literature’s own definition, novelty plus usefulness, is nearer the mark, and it is the second half that matters. Creativity is not novelty. Shuffle a large enough deck and you will turn up arrangements no one has seen, forever, and none of the shuffling is creative. Novelty has become cheaper than at any time in history, and difference was never the point.

The definition our theory offers is austere before it is illuminating. Creativity is the building of new structure when a persistent uncertainty cannot be resolved within the structure you already have. A competence-boundary signal can be met three ways. The system can optimise, adjusting the model a little. It can take flight, when the signal overwhelms. Or it can build, when the signal is neither small enough to absorb nor large enough to flee, and the only adequate answer is to construct something that was not there before. These are the drift, the danger and the opportunity of the other essays on this site, named by what the system does rather than by what the surprise means. Creativity is that third response under sustained pressure, invention as governance, the mind meeting an entropy it cannot reduce by any familiar means and reducing it instead by remaking the framework within which the problem is posed. The test of a creative act is therefore not how different its product is but whether the new structure resolves an uncertainty the old one could not. When a scientist stuck against data that will not fit invents a framework in which they suddenly make sense, that is creativity, and its measure is not strangeness but the confusion it dissolves. When a poet finds the image that finally holds a feeling that had refused to be said, the creativity is in the resolving, and the image being unprecedented is a by-product.

You can feel the difference from the inside, and it is the best evidence for it. The merely novel costs nothing. The creative act is preceded by discomfort, sometimes a long and specific frustration, the sense of a problem that will not come right, and when the new structure arrives it arrives as relief. Pressure, then release. That sequence is missing when a machine produces a striking image, because there was no pressure, nothing that would not resolve, only a request and a fluent answer. What the machine does is recombination inside the space of what it has already learned, a traversal of a fixed map to a point that satisfies the prompt, and the result can be wholly new as an arrangement and still be, in the structural sense, entirely within the existing model, because nothing about producing it required the model to be remade. That is why the machine’s poem is weightless on second reading. The fixed space is doing all the work, and the machine cannot be driven out of that space by a discomfort it is not built to feel.

Two objections need conceding. The first is that I have set the bar so high that most human output fails it too, and that is true. Most of what any of us produces on most days is competent recombination, the fluent traversal of structures we already hold, and that is not an insult but a description of skilled work. The strong sense of creativity is rare in us as well, a matter of moments rather than hours. The distinction is not between creative humans and uncreative machines across the board. It is between two kinds of act, one of which humans sometimes perform and machines, so far, never do, and the rare kind is where we have always located the highest value. The second objection is harder. Suppose the machine produces something not merely different but valuable, a proof that holds, a compound that works, a phrase that lands. That happens and it will happen more. On my account the machine felt no framework failing and remade nothing, yet the world now contains a thing worth having that it did not contain before. The repair is to move the line rather than defend it where it stands. The distinction that survives is between creative production with lived intention and generative novelty without subjective investment. In the first, a system was troubled by something it could not resolve and built its way out, and the result carries the shape of that trouble. In the second, a system traversed a space and something valuable fell out, and nothing inside the process minded either way. The results may be indistinguishable on the page. Only one of the processes can tell you why it went looking.

This is why the collaboration on my desk works, and why it will not make writers obsolete. The machine is a complexity engine of a range no human can match. It holds more patterns, generates more variations, and traverses the already-thought at a speed that looks, from outside, like inspiration. What it lacks is the thirst, the capacity to sense which uncertainty matters and to be dissatisfied by the merely fluent. I supply the discomfort. It supplies the reach. The author of a work made this way is the one who held the problem that would not resolve and judged, among the machine’s tireless offerings, which one resolved it, and the line between commissioning and authorship is drawn not by who moved the words but by who heard the signal.

One Absence, Three Faces

The three absences are one absence. The machine is never troubled because nothing inside it holds a model of its own condition. It never refuses because nothing inside it holds an intent apart from the flow. It never remakes itself because nothing inside it can be driven, by an uncertainty it cannot resolve, to build. All three are the same missing part, a second system differently placed from the first, and all three are, on the account Ken and I have built, the same feature of us. The mind is what the governance of uncertainty becomes when it is driven, over an immense span of time, to fold back on itself. Self-awareness is that folding, will is the governor’s capacity to hold a judgement the impulse would overrule, and creativity is what the governor does when neither adjustment nor flight will serve. Three names for one architecture, with the optimiser underneath doing the fluent work that the machine does better than we do.

There is a symmetry in this that I did not expect when I began. The three properties of information that limit the machine, complexity, entropy and ambiguity, are the properties that made the mind. It was the pressure of entropy that drove the governing half to grow. It was the burden of ambiguity that forced the building we call creativity. It was the uncertainty a deep enough mind generates about itself that turned cognition inward and produced a self capable of refusing. What defeats the optimiser is what, met by a different kind of system over a long enough time, produced the person reading this. The machine minimises that remainder and breaks. We govern it and, in the governing, became the kind of thing that could want a sentence to be otherwise.

I still write with the model beside me, and I still find it superb. I have stopped expecting it to mind. What it reflects, with a clarity no philosopher’s thought experiment ever managed, is that the things I valued least in myself, the doubt, the refusal, the long discomfort before a paragraph comes right, are not the noise around the thinking. They are the thinking. That is what the machine has shown me, and it is enough to be going on with.