The AI-Enabled Masters: Who Wins, Who Loses, and Who Rules

Every certified system I have worked on ended at the same table. The compliance documents were stacked in the order the regulator would read them, the software had done everything the tests could ask of it, and the last thing that happened was that a person signed. Not the company: a person, with a name, whose signature said that they had satisfied themselves about what the software would do, and what it would not do, in the cases the tests had not covered. I have watched that pen hover. The engineers in the room had spent months making the machine right on average. The signature was for the exception, and it could not be delegated to the machine, because the machine cannot sign for what it cannot see. A named human answerable for the edge of a machine’s competence is the oldest idea in my industry, and it is what this essay is about, because almost everywhere outside my industry it is being quietly removed.

In 1765 the Mughal emperor granted the East India Company the right to collect the taxes of Bengal. The Company was not a government. It was a joint-stock enterprise with shareholders in London, a single objective, which was the return on their capital, and no obligation to anyone in Bengal beyond what the objective required. Within five years a famine had killed a large fraction of the province’s people, the Company’s revenue collection continued through it, and the directors in Leadenhall Street were, by every account, surprised at how badly things had gone. They had not set out to be cruel. They had set out to optimise, and the machinery they built for optimising had no part of itself that could register an exception. A province was, to that machinery, an average.

The Company belongs here because the argument is old and the technology is new, and the age of the argument matters. Artificial intelligence does not distribute power. It concentrates it, and it does so by a mechanism that has concentrated power before: it takes a decision that used to be made by many people exercising judgement, reduces it to a rule that a machine can run at scale, and hands ownership of the rule to whoever owns the machine. The Company had ledgers and agents. We have optimisers. The shape is the same, the speed is not, and the question at the centre of it, the one every earlier version of this story turned on, is not whether the rule is fair on average. It is who decides when the rule does not apply.

The Tyranny of the Average

Every system that now touches a life at scale is, underneath its particular surface, an optimiser. It has been shown an enormous number of examples and scored by a single number, and it has learned the patterns that make the number go down. That is a magnificent way to reduce complexity, which is what these machines are for, and the two faces of information they cannot see are the subject of The Three Faces of Information. What matters here is a consequence of the reduction that has nothing to do with accuracy. An optimiser decides in probabilities, not individualities. It is right about the population, and being right about the population is exactly what a person who is unusual cannot use.

Tocqueville, watching American democracy in 1835, worried about a tyranny of the majority, and John Stuart Mill made the phrase famous in On Liberty. Their fear was political: that a majority could oppress a minority through law and, worse, through custom, with no tyrant anyone could point to. Statistical governance is the same fear in a new costume. A system that acts on the average is not oppressing anyone in particular. It is simply optimising for the middle of the distribution, and the person at the edge of it, the applicant with an unconventional employment history, the patient whose condition presents oddly, the family whose finances do not resemble the training data, is not refused by a decision anyone made. He is refused by the shape of the curve. Nobody chose it, nobody can be argued with, and the refusal arrives in the same confident tone as every other output, because the machine has one voice.

The deeper loss is the exception itself. Every system of rules ever devised has been paired with a mechanism for the cases the rules did not foresee, because the people who built the rules knew they were general and the world is particular. Aristotle named it in the Nicomachean Ethics: equity, the correction of law where the law fails through its universality, and he said that the equitable person is the one who does not stand on his rights but takes less than his share. The clerk who waives the fee. The doctor who orders the test the protocol does not cover. The bank manager who knows the family and extends the overdraft. The judge who reads the file. These people are, in the terms I use for machines, the second system, the part of the arrangement positioned to notice that this case is not the one the rule was written for and to act on the noticing. They are expensive, slow, inconsistent, and occasionally corrupt, which is why they are the first thing an optimiser replaces. When they go, the exception goes with them, not because anyone abolished it but because nothing in the new system is capable of seeing one. The optimiser cannot ask whether the rule still applies, for the same reason a ruler cannot measure its own length.

This is what I mean when I say that under statistical governance anyone can become an edge case. You do not have to be poor or foreign or unlucky, though those help. You have to fall, once, outside the pattern the machine was trained on: a name that matches the wrong record, a transaction at an unusual hour in an unusual place, an illness that does not look like the ten thousand illnesses before it. From that moment the system treats you as noise, and it has no mechanism for finding out that you are not. The old world had a clerk. The new one has a form that leads back to the same machine.

The strongest objection to this section is that the clerk was not a hero. The clerk who waived the fee waived it for people who looked like him. The doctor who ordered the extra test ordered it, the studies say, less often for women and for black patients. The bank manager who knew the family knew the families he was disposed to know, and redlining was not an algorithm’s invention; it was drawn by hand. Jon Kleinberg and his colleagues, in their 2018 study of bail decisions in New York, found that a model could have cut the jail population by more than 40 per cent with no rise in crime, or cut crime by a quarter at the same jailing rate, and reduced the racial disparity in the process, because the judges’ discretion had been doing worse than the average on both counts. I accept all of it. The historic engines of discrimination were people exercising judgement, and an optimiser that is right about the population is often fairer to the population than the clerk was. That is an argument for the rule, and I have made it myself. It is not an argument against the exception. The bail model that beats the judges on average still meets the defendant whose circumstances it has never seen, and the question of who decides that case has not gone away because the judges were biased. It has been answered, silently, in favour of nobody. What the record of the clerk shows is that the exception has to be accountable, which it seldom was. It does not show that the exception can be abolished.

Who Decides the Exception

The jurist Carl Schmitt, whose later politics were vile and whose sentence survives him, wrote in 1922 that sovereign is he who decides on the exception. He meant that the real location of power in any order is not in the rules but in the authority that can suspend them, and the observation is sharper now than it was in Weimar. The rules that govern credit, employment, insurance, information and, increasingly, the attention of an entire population are run by optimisers. The power to decide the exception, to say that this case is not what the rule was for, belongs to whoever owns the optimiser, and that is not, for the most part, a government. It is a small number of companies, and within them a smaller number of people who set the objective the machine is scored on.

The usual framing, corporations against the state, misses where the power sits. The Dutch tax authority that ruined tens of thousands of families with a benefits-fraud model was a state. The English examinations regulator that downgraded a generation’s grades by algorithm was a state. Power does not concentrate in corporations as such. It concentrates in whoever owns the objective, and the state has proved as willing to hand its exceptions to an optimiser as any firm, and less able to explain itself afterwards, because it bought the machine from someone else. What is different about the private case is only that the objective is not even nominally the public’s. A feed optimised for engagement, a hiring tool optimised for resemblance to past hires, an insurer’s claims engine optimised for cost, each decides the exception according to a number nobody outside the company chose, and the people the number falls on cannot vote it out.

The concentration has a second layer that the East India Company would have recognised. The machinery of optimisation at the frontier, the models, the data and the computing plant to train them, is owned by a handful of firms, and everyone else, including governments, rents it. A company that rents its optimiser does not own the objective either. It inherits the biases, the failure modes and the release schedule of a supplier it cannot see inside, and when the supplier ships a new version, every decision downstream changes in ways nobody in the chain has audited. Daron Acemoglu and Simon Johnson, in Power and Progress, trace a thousand years of technology and reach a conclusion that the industry’s optimism does not like: the gains from a new technology go to whoever controls it unless some countervailing power forces them to be shared, and there is nothing automatic about the sharing. The medieval plough enriched the lords. The early factory enriched the mill owners for two generations before the workers saw a rise in wages, and the rise came from unions and the franchise, not from the machines. The pattern is not a law of nature, but it has held often enough to be the default.

Control Without Coercion

What makes this concentration unlike its predecessors is that it requires no force. The Company needed sepoys. The mill owner needed the magistrate. An optimiser needs only that we trust it, and we do, because it is convenient, because it is usually right, and because the person it is wrong about is, by construction, somebody else. Nobody is compelled to accept the feed’s ordering of the world or the lender’s ordering of the queue. They are simply the default, and the default is where most lives are lived. Control that arrives as a recommendation is harder to resist than control that arrives as an order, because there is nothing to refuse. There is only a screen, and the screen is helpful.

This is most visible in the shaping of what people see and believe, and the mechanism is well documented. A system that decides which 40 of yesterday’s 10,000 events you are shown is not manipulating you. It is optimising for the number it was given, and if attention is the number then outrage is what the optimisation produces. The point for this essay is a narrower one. The power to surface some ideas and bury others used to be distributed across editors, publishers, teachers, pulpits and pubs, each with its own biases, and the biases cancelled in the aggregate because there were so many of them, or so the argument runs; whether they cancelled or merely agreed is contested, and the old order had orthodoxies of its own. What is not contested is the count. The power is now held by a few systems with one objective each, and whatever the biases do, they no longer have many rivals to do it against.

The Digital Oligarchy

Put the layers together and the shape is an oligarchy of a new kind, and I use the word with some care, because it is easily misheard as a conspiracy. There is no conspiracy. There are a few firms with the capital to build frontier optimisers, a much larger number of institutions, public and private, that rent them to run their rules, and a population that receives the outputs. Nobody in the chain has to intend harm. The harm is a property of the architecture: an objective set at the top, run at scale, with no part of the system positioned to see the exception, and no person at any level who both bears the consequence and holds the power to decide.

The precedents are instructive because they ended. The East India Company was brought under parliamentary control by degrees after Bengal and dissolved in 1874. Standard Oil, which had reduced the complexity of American energy into a single ledger, was broken into 34 companies in 1911 under a law written to contain exactly that concentration. The mill owners met the trade union and then the ballot. In every case the technology that concentrated the power was not abandoned; the railways still ran and the oil still flowed. What changed was the location of the exception. Somebody was made answerable for the cases the machinery could not see, and the machinery was left to do the work it was good at.

That is the form the answer takes here too, and it is not a demand for less AI. The optimiser is the best complexity-reducing instrument ever built, and a society that refused it would be poorer and would still be governed by averages, only slower ones. The answer is to put the exception back, and to put it where the consequence is. Three things follow, and I hold each with the doubts left in. The first is a right to a human decision on the cases the machine itself should have flagged, defined by the system’s own uncertainty rather than by the persistence of the person complaining, because the machinery to identify those cases exists and its absence is a choice. Article 22 of the European data-protection regulation already grants a version of this, the right not to be subject to a decision based solely on automated processing, and its weakness is instructive: it is triggered by the person objecting, not by the machine flagging, so it protects the persistent rather than the unusual. The second is that ownership of an objective carries answerability for its exceptions: an institution that runs a rule by optimiser signs for what the rule does to the people it did not foresee, whether it built the machine or rented it. The third is countervailing power of the old-fashioned kind, competition among the owners of the optimisers, open alternatives to the closed ones, and a public willing to make the exception a political question rather than a customer-service one. None of these is guaranteed. Every earlier version of this story took a generation and was won by people who organised, against the orderly objection of those who owned the machines.

Who Rules

I have spent my working life building systems that have to be right, in a discipline where a named person signs for what a machine will and will not do, and I am not a pessimist about the technology. I am a pessimist about the arrangement, because the arrangement is old and I know how it ends when nobody intervenes. The question the title asks has an answer, and it is not the one the industry gives. The winners are the owners of the objective. The losers are the exceptions, and anyone can become one. Who rules is whoever decides when the rule does not apply, and at present, in most of the systems that touch a life, nobody does, which is the same as saying the average rules and the edge is on its own.

The Company’s directors were not monsters. They were shareholders who had built a machine for collecting revenue and had never asked what it would do in a year the rains failed. Ours are not monsters either. They have built a machine for reducing complexity and have not asked, or have not been made to ask, what it does to the person the complexity was reduced away from. That question was answered once by a parliament, once by a court, once by a picket line. It will have to be answered again. The machines will not answer it; that is not what they are for.