The rise of AI consumerism: Living in a world of invisible algorithms

My business bank account reads my supplier invoices now. I upload a document, and before I have finished reading it the payment screen has been filled in, the vendor, the amount, the reference, all correct more often than I would manage on a tired Friday afternoon. I check every one. I check because the money is mine, because the signature on the payment is mine, and because I have spent enough of my working life around machines that are confidently wrong to know that the tidy screen tells me nothing about whether this is one of those days. The checking takes ten seconds. The machine has saved me ten minutes. That is a good trade, and it is the shape of the best of what these systems do for us, small, constant, and supervised by the person who bears the consequence.

Most of the algorithms that touch a life are not like that. Somewhere this week a system decided which résumés a recruiter would see and which would never reach a human eye. Another decided whether a family’s application for credit went to a person or into a queue. Another scored a claim for medical treatment, and another scored a defendant for a judge who had eleven minutes for the case. Another chose which 40 of the 10,000 things that happened yesterday you would be shown this morning. None of the people on the receiving end saw the machine. None of them was told a machine was involved. Nobody checked, because nobody’s signature was on the outcome, and the person who bore the consequence was the one person with no view of the screen. This essay is about that asymmetry, and about the question I have come to think sits underneath every worry about algorithms in ordinary life, which is not whether the machine is good or bad but whether anyone in the chain can tell which kind of machine it is.

The Optimiser in the Room

Almost every system in this story is, underneath its particular surface, one mechanism. It has been shown an enormous number of examples and scored by a single number, then adjusted, billions of times, until the number is as good as it will go. For a feed, the number is engagement. For a lender, it is predicted default. For a recruiter’s tool, it is resemblance to the people who were hired before. For a hospital’s triage system, it is predicted cost. The systems are not evil and they are not stupid. They are magnificent at finding pattern in the part of the world that has pattern, and they are used because they are cheap, fast and, on average, better than the tired human they replaced.

The trouble with them is not a matter of degree, and Ken Wenger and I have spent several years working out exactly what it is. Information has three faces, complexity, entropy and ambiguity, which I have set out in the primer The Three Faces of Information, and a single-number optimiser sees only the first. A system permitted only one output must collapse an ambiguous case, one appearance consistent with several causes, into a single answer, and it does so with the same confidence it wears everywhere else, because nothing inside it is positioned to notice that a choice was made. Two applicants with identical files and different futures. Two patients with the same bill and different needs. Two posts with the same engagement and opposite effects on the person reading them. To the optimiser these are the same event, and the fork in each is flattened before anyone could see it was a fork.

That is worth holding onto, because the popular story about algorithms is a story about manipulation, and manipulation is not what is happening. The feed is not trying to make you angry. It is optimising for the one number it was given, and anger keeps people scrolling, so anger is what the collapse produces. The hiring tool is not prejudiced. It was scored on resemblance to a past, and the past was prejudiced, so the resemblance is too. These systems have no intent. They have an objective, and an objective pursued relentlessly by something that cannot ask whether the objective still applies is a far stranger thing to live under than a villain, because there is no one to argue with.

Computer Says No

The documented cases are worth setting down plainly, because each one shows the collapse from a different angle and none of them is exotic. In 2018 it emerged that Amazon had spent four years building a tool to rank software candidates and had abandoned it after discovering that it marked down any résumé containing the word “women’s”, as in a women’s chess club, and downgraded graduates of two all-women colleges. Nobody had told it to. It had been trained on a decade of hiring in which most successful applicants were men, and it had learned, faithfully, what success looked like. The following year a study in Science examined a system used by American hospitals to decide which patients needed extra care, a system applied to something like 200 million people a year. It used cost as a proxy for need, and because less money had historically been spent on Black patients with the same conditions, it rated them as healthier at the same level of illness. Correcting the proxy would have more than doubled the share of Black patients flagged for help. The system was not wrong about cost. It was asked the wrong question, and it had no way to know.

In 2020 the English examinations regulator, unable to hold exams during the pandemic, used an algorithm to assign A-level grades from teachers’ assessments and each school’s past results. Nearly two in five grades came out lower than the teachers had predicted, with the sharpest downgrades falling on bright pupils at schools with poor histories, because the model had learned that such schools do not produce top grades and applied the lesson to the individuals in front of it. The results were withdrawn within four days, after teenagers stood outside the Department for Education with placards. In the Netherlands, the tax authority’s risk model for childcare benefits fraud flagged tens of thousands of families, many of them for holding a second nationality, and demanded repayment of sums that ruined them. Some lost their homes and some their children. In January 2021 the whole government resigned over it. In the American courts a commercial tool that scores defendants for the likelihood of reoffending was found, in a 2016 investigation, to wrongly flag Black defendants as future criminals at almost twice the rate of white ones, while the company defended its accuracy on the average and the judges who used it could not see inside it.

Kafka wrote the definitive account of this experience in 1925, before there was a machine to blame. Josef K. is arrested one morning without being told the charge, is processed by a court whose rules he cannot learn, and is never in the whole of The Trial granted the one thing he asks for, which is to know what he has done. The horror of the book is not cruelty. It is procedure without a person inside it. That is the texture of a rejected application with no reason attached, a denied claim with an appeal form that leads to the same machine, a grade that fell because of a school’s history. The person who bears the consequence cannot find anyone who decided.

Who Signs?

The question I keep returning to is not the one the field usually asks. The field asks whether the model is accurate. I ask who signs. I spent the better part of two decades building safety-critical software for aircraft, and in that world the question of accuracy comes second. The first question is whether a system can be shown, to a regulator, never to do the thing it must not do, and the second is whose name is on the document that says so. A cockpit display does not ship because it is usually right. It ships because a named engineer has signed that its behaviour is bounded, and if it is not, that engineer answers for it. Nothing like this exists for the systems deciding whether a family gets a loan.

Look at the small line of text beneath the box where you type a prompt into any of the current assistants. It says, in one wording or another, that the system can make mistakes and that you should check important information. That is not boilerplate. It is the business model. The accountability for whether the answer is right has been handed back to the user, and the machine is, in the strict sense, subjective: not responsible for the correctness or consistency of its own output. That arrangement is tolerable when the user is me checking an invoice, because I hold the consequence and I can check. It becomes something else when the same kind of system is placed in a decision where the person bearing the consequence has no view, no expertise and no appeal. The disclaimer is still there. It has simply moved to a place where nobody who is affected can read it.

Two cases show what happens to responsibility when it has nowhere to land. In February 2024 a Canadian tribunal found Air Canada liable for its own customer service chatbot, which had invented a bereavement refund policy and told a grieving passenger to book first and claim later. The airline argued that the chatbot was a separate legal entity responsible for its own statements. The tribunal rejected this, and the argument deserves to be remembered, because it is the accountability gap stated in the open by a company that thought it sounded reasonable. The other case is quieter and more instructive. In 2022 the American insurer Cigna was reported to be running a system that matched claims against a list and flagged mismatches for denial, letting company doctors reject batches with a single electronic signature without opening the files. Over two months it was used to deny more than 300,000 claims at an average of about 1.2 seconds each. The company disputed the characterisation. What is telling is the arithmetic underneath. Only a small fraction of people ever appeal a denial, and where appeals were filed, most were overturned. Read those numbers together and you see a process designed around the assumption that wrong outputs would mostly go uncontested. The mistakes were not an accident the system tolerated. They were a cost it had priced, against the near-certainty that most people would pay the bill rather than fight.

That is what a society governed by unseen decisions looks like from the inside. Not a dystopia, and not a conspiracy. An accounting, in which the errors of a subjective machine are charged to the people least able to detect them, because those people are cheaper than a second system that would catch the errors first.

The Wrong Complaint About the Black Box

The remedy everyone reaches for is explainability. Open the box, the argument goes, make the model say why, and the arbitrariness will drain out. I have some sympathy, and a great deal of scepticism, because the complaint is aimed at the wrong absence. A deep network does not reason and then hide the reasoning. It arrives at its output by a route that is not a reason in any sense a person could act on, a weighted sum across a million features, and an explanation extracted from it after the fact is a story told about the machine rather than the machine’s own account. Demand explanations and you will get them, fluent ones, and they will be no more reliable than the decision they are explaining. The machines have become superb at producing plausible reasons. That is the problem, not the solution.

What is missing from these systems is not a window. It is any part of the machine positioned to know when it does not know. An optimiser cannot ask whether optimising is still the right thing to do, for the same reason a ruler cannot measure its own length. The question requires a vantage point the optimiser does not contain. Ken and I have spent the last several years building that vantage point as a second system, and the finding that matters most for this essay is a modest one. A trained model holds more in its internal geometry than its final answer reveals. Read it one step before it commits, and you can see whether an input sits on familiar ground, out at a boundary, or between two clusters. On handwritten digits that reading cut the errors on the most confusable numeral by three-quarters without touching the underlying model. On an ordinary object detector it sorted the cases into trusted, ambiguous and unfamiliar, and the sorting was worth more than the accuracy, because the system knew which cases were hard. That is the capability a consumer needs from the machines deciding about them, and it has nothing to do with explanation. It is the machine’s ability to say, of this application, this claim, this defendant, that it is one of the ones on which it should not be trusted, and to hand that case to a person with the two or three readings it could not choose between.

This is not a lonely proposal. The machine-learning literature on learning to defer builds classifiers that decide when to answer and when to hand a case to a person, and out-of-distribution detection flags the inputs a model has never seen the like of; both exist, both work within their limits, and either would have done more for the pupils and the patients than an explanation. The difference is in what triggers the deferral. A deferral rule trained on past errors learns where the model used to be wrong; a novelty detector learns what the model has not seen. Neither reads the case that is ordinary, in distribution and ambiguous, the claim that looks like a thousand honest claims and a thousand dishonest ones, and that is the case the harm falls on. The second system reads it, because it reads the geometry of the model’s own state rather than the model’s history or the input’s strangeness.

The A-level algorithm could not do that. Every pupil at a school with a poor history looked the same to it. The hospital model could not do that; the patients whose bills understated their illness were not flagged as ambiguous, they were flagged as well. The benefit fraud model could not do that. In every case the harm fell on exactly the people a second system would have marked as the cases the first system was least entitled to decide. The right to an explanation is a weak right, because the explanation can be manufactured. The right to a human on the cases the machine itself should have flagged is a strong one, because it can be engineered, tested and audited, and because the machinery to identify those cases exists. Article 22 of the European data-protection regulation already grants a right not to be subject to a decision based solely on automated processing, and it has been a weak right in practice for the same reason the right to an explanation is weak: it does not say which cases, and a right to a human on every case is a right nobody can afford to honour. Tie it to the cases the machine flags and it becomes enforceable.

Two Kinds of Machine

Not all AI is unreliable, and any account that says otherwise should not be trusted. A model reading mammograms in a screening programme, tested against verified outcomes, bounded to one task, with a radiologist reading every case it flags and a sample of those it does not, is a different kind of object from a résumé filter bought from a vendor on a demonstration. The difference is not that one is cleverer. It is that one operates inside a defined range with a person at the decision points and a record of where its errors cluster, and the other operates on whoever comes through the door, at scale, unwatched. I would call the first objective AI, in the sense that it is accountable for its own correctness within a stated boundary, and the second subjective, in the sense that its correctness is somebody else’s problem, and the somebody else is you.

The consumer cannot tell them apart, and that is the whole of the difficulty. Both arrive as a screen. Both are confident. The bank’s invoice reader and the insurer’s denial engine look identical from the outside, and only one of them has a person checking who bears the consequence. Regulation is the usual answer, and some of it will help. The European Union’s approach, which ranks uses of AI by the risk they pose to the person on the receiving end rather than by the cleverness of the model, is the right shape of idea, and it is a different shape from the American and Chinese race to own the technology. Rules, though, are slow, and a rule that requires an explanation will get one. Three things would do more, and none of them requires a new model.

The first is disclosure of the plain fact. A person should know when a machine made or materially shaped a decision about their employment, their credit, their treatment or their liberty. Not how, which they could not use, but that. The second is the human on the hard cases, defined not by the applicant’s persistence but by the machine’s own signal, so that the cases routed to a person are the ones the system was least entitled to decide rather than the ones belonging to whoever knew how to complain. The third is audit by error cluster rather than by average. Every one of the failures above was invisible on the average. The Amazon tool was accurate on the average. The hospital model was accurate on the average. The errors lived in identifiable regions, women’s colleges, poor schools, second nationalities, and a system that reports its average accuracy while its errors gather in one corner of the population is reporting the one number that hides the harm. An audit that asks where the mistakes cluster is asking the question the geometry answers, and it is the question a regulator with a certification background would ask first.

I hold these proposals with the doubts left in. Disclosure can become a checkbox. A human on the hard cases can become a rubber stamp, as the doctors with the single electronic signature became. Audits can be gamed by whoever defines the clusters. None of that is a reason not to do them. It is a reason to remember that the thing being governed is not a machine but a chain of people who have each, for good local reasons, declined to sign.

The Signature

I still check the invoices. I will keep checking them after the machine has been right a thousand times in a row, because the thousandth time tells me nothing about the next, and because the checking is not distrust of the tool. It is the part of the arrangement that makes the tool safe to use. A subjective machine supervised by the person who bears the consequence is one of the best things the last decade has produced. The same machine placed between an institution and a stranger, with the disclaimer moved out of sight and the errors charged to the stranger, is one of the worst, and the two are built from identical parts.

The question to ask of any algorithm that touches a life is not whether it is fair, which is a property no one can read off a screen, and not whether it can explain itself, which it will do fluently and unreliably. It is the question my industry asks of every system it lets near an aircraft. Does it know the edge of what it can decide, and who signs for what happens beyond it? A machine that can answer the first is worth trusting on the ground it knows. A chain of people who can answer the second is the only thing that has ever made a decision accountable. Where neither answer exists, the person on the receiving end is Josef K., and the fact that the court is made of arithmetic rather than clerks does not make the charge any easier to learn.