Optimization

In Chapter 1 we saw that a well-chosen model can compute whatever we want, provided we set it up with the right parameters.

But how do you find the right parameters?

The answer is to use the same method that drives evolution: make a tiny change, check whether it made the model work better, keep it if it did, and repeat this process very very fast, many many times.

Nothing Complex Was Ever Designed From Scratch

When we see something complex and impressive, it's tempting to assume someone brilliant sat down and designed it. But that's almost never how complex things actually come to exist. Instead, pretty much everything complex in the world was created through a long sequence of incremental optimization steps.

AI is no exception. But to make this work, you need two things: a way to measure how far off you are, and a way to make small changes.

You Need a Way to Measure the Error

Before you can improve anything, you need a number that tells you how far off it is. This number is called the error: the gap between where you are and where you want to be.

This idea shows up everywhere. A student gets 72 out of 100 on a test. The error is 28 points. A runner finishes in 14.2 seconds. The error is 14.2 seconds. A restaurant has 3.2 stars. The error is the gap to a perfect 5.

In AI, we do the same thing. We show the model an input, look at what it gives back, and compare that to the right answer. The gap between them is the error. The goal of training is to make that error as small as possible.

The Algorithm That Built the World

Now we can be precise about the pattern. Call it incremental optimization:

1. Measure the error
2. Make a small change
3. Did the error go down? Keep it. Did it go up? Undo it.
4. Repeat

That's it. And this single algorithm is behind:

• Natural selection: random mutations, tested by survival, kept or eliminated
• The scientific method: hypotheses, tested by experiment, accepted or rejected
• Trial and error: try something, check whether it's better, adjust

This is how AI learns. Small adjustments to parameters, tested against data, kept if they reduce the error.

What's striking about this approach: you don't need to understand the system. You don't need to know why a wing shape generates lift, or why a particular word order sounds natural, or why certain parameter values make a model work. You just need to measure the error and make small changes.

Try this out in the playground yourself. See how much easier it is to find the best answer using incremental changes guided by an error signal, than if you have to just make blind guesses.

Smooth Functions Can Be Optimized

Incremental optimization is powerful, but it only works when the thing you're optimizing is smooth, meaning small changes to what you do produce small, predictable changes in the error. Smoothness is what gives you feedback: each small adjustment tells you whether you're getting warmer or colder.

Not everything works this way. Some things are all-or-nothing. A small change gives you no useful information at all.

Both shapes below have the same overall valley, but the smooth one lets a ball roll downhill to the bottom, while the stepped one is flat within each step. A small change in position tells you nothing.

The Gradient Trick

So far, our optimization strategy has been: try a small change, measure the error, keep it if it helped. That works, but it's slow, because you're guessing which change to make.

Think about the difference between these two situations. A friend reads your story and says "I'd give it 3 out of 5." That tells you the error, but not what to fix. Now imagine they say "I'd give it 3 out of 5, the middle section was confusing." That's much more useful, because it tells you which direction to change.

This is the key idea behind the gradient. At any point on a smooth curve, you can look at the slope: the steepness right where you're standing. The slope tells you exactly which direction is downhill. Following it is called gradient descent, and it's dramatically faster than random guessing.

Real models don't have just one knob; they have millions. But the same idea works: the gradient is an arrow pointing in the steepest downhill direction across all dimensions at once. You can't visualize a million-dimensional landscape, but the math is the same. Calculate the slope, step downhill, repeat.

Why This Changes Everything

In Chapter 1, we saw that a model is a machine with adjustable knobs. The question was: how do you find the right settings when there are millions of knobs?

Now we have two powerful ideas:

• Smoothness means small changes to the knobs produce small, predictable changes in the error. Every tiny adjustment gives you useful feedback: did the error go up or down?
• The gradient tells you which direction to adjust every knob at once. Instead of guessing, you calculate exactly which way is downhill.

Together, these make optimization fast and reliable. No single adjustment is impressive. Each one is a tiny tweak, make this parameter 0.001 bigger, that one 0.002 smaller. But each tweak reliably moves in the right direction. Make billions of these tiny improvements per second, and the model develops amazing abilities.

It's like evolution, compressed from billions of years into hours, by making each step reliable and then taking a huge number of them.

AI didn't take off when people figured out the theory; the basic ideas have been around for decades. It took off when there was enough data to measure error on, enough computing power to make billions of tiny adjustments quickly, and the right kind of model: one where the knobs are smooth and the gradient always points the way.

What's Next

We now understand the game:

1. Build a smooth model (so small changes give useful feedback and gradients work)
2. Measure how wrong the model is (the "loss function")
3. Calculate which way is downhill (the gradient)
4. Take a tiny step (adjust all parameters)
5. Repeat billions of times

But what should this model actually look like? It has to be smooth, so the gradient always points the way. And it has to be able to compute any function we might want, or no amount of training will find it. The cubic from Chapter 1 is smooth but can't do much. In Chapter 3, we'll start from the logic gates inside every computer, which can compute anything but aren't smooth, and turn them into something that is both: the neural network.