Reasoning

Ask a person a hard question and demand an answer in the next half second, and they will do badly. Not because they cannot solve it, but because they need time to think through the problem.

A transformer has the same problem. Each token gets a fixed amount of computation, a fixed number of layers. A question needing ten steps of work does not get ten times the compute just because it is harder.

Writing To Itself First

If you ask a human to solve a hard problem, they might talk to themselves inside their head, or get out a pen and paper and write out their reasoning.

Reasoning models solve this problem in roughly the same way. The model writes out its thinking in a series of hidden thinking messages that the human never sees, and only then writes its reply.

Nothing about the architecture changes. There is no new component and no extra network. The thinking tokens are predicted exactly like every other token, and the model reads them back the same way it reads anything else in its context. Its own output becomes its scratch paper.

How They Are Trained

The obvious way to teach this would be to write out ideal reasoning and train on it, the way Chapter 10 showed for ideal answers. That approach is useful, and there is a lot of material for it, since the web is full of homework help and worked textbook solutions. What it does not cover is breadth and difficulty. The problems people write up are the standard, well-trodden ones, and the hard or unusual problems where reasoning has to be good are exactly the ones nobody has written up.

AI models learn to reason in pretty much the same way humans do. In a classroom, a teacher gives students problems to solve whose answers can easily be checked for correctness. The students try different ways of reasoning until they learn the methods that quickly lead to an answer that gets marked correct.

We do essentially the same thing for an AI model by giving it many problems whose answers can be easily checked, letting it try to solve each problem many times over, and reward the outputs that quickly lead to a good answer.

Repeat that over millions of problems and the habits that survive are the ones that lead to correct answers: checking your work, backing out of a dead end, noticing halfway through that an earlier step was wrong.

Maths and code improved most, because those are the domains where it's easiest to show that an answer is correct. But the method is not limited to them. Judging something is easier than producing it almost everywhere: it is much easier to tell that a poem is good than to write one, and much easier to tell that a business plan is good than to come up with it. So the same training works in fuzzier subjects too, though not as effectively.

Two Things To Keep Straight

The hidden thinking messages are not a complete account of what drove the answer. Give a model a hint of what answer you want, and it will often write a confident chain of reasoning that justifies the answer it thinks you want, without ever mentioning your hint. People do the same thing.

More thinking is also not always better. Every thinking token costs time and money, and on an easy question a long stretch of thinking can give the model more chances to talk itself out of the right answer.