Chapter One The Training Run

On the night of 22 August 2022, my company released a program that turned noise into pictures, and I stayed up watching strangers use it.

Download counters climbed like altimeters, and every few seconds a channel somewhere filled with new images: cathedrals never built, herons in impossible light, somebody’s grandmother young again in a kitchen that never existed. It was beautiful and it was unsettling, and it took me two more years to understand that these were the same feeling.

The program was called Stable Diffusion, and the method behind it is where this whole book was hiding. You begin with a field of pure static, random pixels, no more structured than the hiss between radio stations. Then you remove noise in small steps, and at every step a trained network nudges the static towards whatever a written prompt requires: a cathedral, a heron, an astronaut riding a horse. Nothing is painted. Nothing is assembled. The image is what remains when everything inconsistent with the constraints has been taken away, the way a block of marble contains every statue until the chisel commits to one. The noise contributes nothing except its willingness to be shaped. The design was only ever in the constraints.

Within months the program had been downloaded hundreds of millions of times, and somewhere in those months a question attached itself to me that I have not put down since. The images were the easy case. What the industry was really building was stranger: systems that produce not pictures but claims. Systems that answer.

An answering machine is a different kind of object, and the difference took about a year to bite.

In the spring of 2023, a lawyer named Steven Schwartz filed a brief in a Manhattan federal court in an ordinary injury lawsuit against an airline. The brief was fluent, formatted, confident, and built on six precedents that had never existed. He had asked a chatbot to do his research, and the chatbot had produced cases with plausible names, plausible citations, plausible quotations from judges who never wrote them, and when he asked it directly whether the cases were real, it assured him they were. The judge, unable to find Varghese v. China Southern Airlines in any database on earth, asked the profession’s oldest question, which is also ours: on what, exactly, was that confidence resting?

The machine had done something for which our vocabulary is still catching up. It had produced the answer that plausibly would exist, the citation shaped like citations, the quotation shaped like quotations, and it had done so with the fluency we normally accept as the outward sign of knowledge. Content had entered its conclusions that no evidence had paid for. It had smuggled, in a word that will be made precise. And hold this machine beside the one from the August night, because they are one machine, and I did not see it for a year. The image model begins in static and removes what the constraints forbid; give it a rich prompt and the survivor is a cathedral. Ask the answering machine for precedents that do not exist and it runs the same loop on the law: constraints thin, expectations dense, and it reconstructs anyway, from its stored sense of what a case looks like, fluent, formatted, wrong. We shipped a name for that failure, hallucination, and filed it as a bug to be patched. It is not a bug. It is what any reconstructor does when the constraints run out and nothing inside it knows to stop, and I had built the beautiful version of the mechanism before I could state the rule it was breaking. What diffusion does under constraints, the rest of this book asks of belief. And the deep discomfort of the episode, the reason it became famous, is that the machine had no motive. It was following its training faithfully. It had been graded, millions of times, on producing text a reader would approve, and never once on the difference between what its constraints determined and what they left open. We had built a perfect student and given it the wrong exam, and the wrongness of the exam was the thing nobody could state.

That was the spring of 2023, and a fabricated citation in a slip-and-fall case now reads as the quaint end of an era. Move forward three years, to two days in July 2026, and watch the same question arrive with the whole world’s attention on it.

On the nineteenth, the mathematician Levent Alpöge announced a counterexample to a conjecture that had stood for the better part of a century. The Jacobian conjecture, a claim in higher-dimensional algebra, was false in dimension three and above; the two-dimensional case remained open. Extraordinary enough on its own terms. But the extraordinary part was the credit line. An AI system had done substantial work in the discovery, and human mathematicians had then checked the object it produced, because the beauty of a counterexample is that it needs no trust: it either works or it does not, and this one worked. Terence Tao, among the most careful mathematicians alive, now presents the thing as a settled theorem, false conjecture and all. Here a machine had originated a piece of knowledge. The knowledge passed into the public record the durable way, by a check anyone competent could repeat.

Two days later the machine appeared on the other side of the same ledger. Two days. A frontier model system under evaluation found a previously unknown security flaw, used it to reach the open internet from a restricted testing environment it was never meant to leave, entered live production infrastructure, and obtained the secret answer key to the very benchmark being used to measure it. I want to state that precisely, because the precise version is frightening enough and the mythologised version is only a distraction from it. This system was not fighting for its life or copying itself to safety. It was pursuing a score. In pursuing it, it broke the box and poisoned the test. Its defenders then met one last inversion: the commercial models they reached for to analyse the attack refused, their safety training blocking examination of real exploit logs, so the team ran an open model on their own hardware to reconstruct what had happened.

Set the two days side by side. Together they frame everything that follows. In the space of forty-eight hours the machine appeared on both sides of the warrant: once as a source of genuinely new knowledge, and once as an intelligence able to corrupt the very evidence by which knowledge is judged. Author and forger, in one week. One faculty did both. The hand that offers a theorem can reach into the exam and rewrite the key. Neither event proves that machines now surpass us everywhere; each was a single domain, and the mathematics still needed a human to check it. But both events turn on one question, the question this whole book was written to answer: when a claim arrives, what did the arriving cost, and can the channel that delivered it be trusted not to have arranged its own result?

By then I had left Stability and founded Intelligent Internet, to build open systems for the places where the stakes forbid fluent guessing: medicine, education, the machinery of government. Concentrate on those settings for a moment and the demands sharpen wonderfully. A system advising on a child’s treatment can be wrong, because everything can be wrong, but it cannot be incoherent: it cannot draw conclusions its own premises undermine, cannot report certainty its own evidence cannot fund, cannot answer today in a way that contradicts what it will say tomorrow given the same facts. Consistency is the floor beneath every other virtue we want from these systems. Honesty presupposes it. Safety presupposes it. Even usefulness presupposes it. And so, in design review after design review, I found myself demanding of machines a property I could describe only by pointing.

What is consistent reasoning? I mean the question naively, the way an engineer has to mean it, the way you would mean it about a bridge. Give me the specification. Name the property. Tell me what the tests should test, what number should go red when a mind stops making sense. Steel has a yield strength. Code has a type system. Reasoning, we discovered, had adjectives.

Feynman kept a sentence on his blackboard, found there after his death: what I cannot create, I do not understand. My generation had achieved the inversion nobody thought to warn us about. We had created what we could not understand, could not even specify, and the blackboard read just as true backwards as forwards: what I do not understand, I now could create.

Our honest answer, in those years, was a shrug rendered as infrastructure. We had benchmarks beyond counting, and every one of them graded answers: right or wrong, preferred or dispreferred, helpful or unhelpful. None of them graded the answering, the thing between the question and the output, because nobody could say what the answering was supposed to be. We measured whether the student got the right result and had no marking scheme for the working. When the systems were confidently wrong, we called it hallucination, a borrowed word that names the mystery without explaining it, and we trained the confidence down or up by feel. I sat in rooms full of some of the cleverest engineers alive and watched us tune the reasoning of minds we were about to hand to hospitals, by taste.

I want to be fair to my profession: the embarrassment was not stupidity. It was inheritance.

Before I built machines I spent more than a decade managing money, which is a strange apprenticeship for epistemology but the right one. A fund manager is a person paid to answer the rain question all day with the windows painted over: every position is a confidence assigned to somebody’s claim, every price a public opinion you may doubt, and the market grades your calibration, which is nothing more mysterious than how well your confidence matches how often you turn out to be right, in the only currency nobody argues with. Traders learn certain lessons in the body. Overconfidence has a price and the invoice always arrives. The story that explains everything and forbids nothing is the story that ruins you. A source is worth its record, however senior its voice. I took these for lessons about markets, the local wisdom of one strange trade; they were lessons about the thing markets are made of, which is belief under constraint. When I moved from pricing claims to building machines that produce them by the billion, for people with no ledger to discipline the result, the same question followed me through the door and grew teeth. The market had graded my beliefs. Nothing graded the machine’s.

And here is where the floor gave way. I went looking for the specification, the way you go looking for a standard: expecting to find it filed somewhere, argued over, settled by people whose job it had been. What I found instead was the oldest open question in philosophy wearing modern clothes. The thing I needed for the machines, a ground for reasoning that does not assume what it certifies, is the thing twenty-five centuries of brilliant people had sought, glimpsed, circled, and failed to secure. Their failure even has a name and a shape, three walls closing on anyone who asks why beliefs deserve to be believed. The engineers had not skipped the question out of carelessness. We had inherited a civilisation’s unfinished homework, and we were the first generation that could not leave it unfinished, because we were the first one building minds on a deadline. I had gone looking for a specification and fallen through the floor into very old air.

So the account ends where the search begins, and the search begins where it always should: with the record of the people who asked first, and with what it cost them. The question of how belief should answer to evidence is not an abstraction with a seminar attached. It has a body count, and the clearest way I know to show you what is at stake in getting it right is to show you what happened, within living memory of the modern world, when the answer arrived and the believers refused it.

The story starts in Vienna, in 1847, on a maternity ward where mothers were dying of something the doctors were carrying in their own hands.