Neural Networks

In Chapter 2 we saw that, if we have a suitable model, we can optimize its parameters to produce whatever function we want - whether that function translates text, writes code, generates images, or anything else.

For this to work, the model needs to have two properties:

• It must be a universal approximator - This means that the model can compute any function of finite input, provided you make it big enough and find the right parameters (Chapter 1).
• It must be smooth - This means that a tiny change to its parameters makes only a tiny change to its output (Chapter 2).

The most widely used model that has both properties is the neural network. This chapter builds one from scratch, starting with logic gates, which can build any function but aren't smooth.

Logic Gates Can Build Any Function, but Aren't Smooth

Every computer is built from logic gates: tiny components that combine true-or-false values, written as 1 and 0.

• AND: "Are both things true?" You need an umbrella if it's raining AND you're going outside.
• OR: "Is at least one thing true?" You'll get wet if it's raining OR someone sprays you with a hose.
• XOR (short for "exclusive or"): "Is exactly one thing true?" "You can have cake or ice cream" usually means one or the other, but not both.
• NOT: Flips the answer. NOT true is false, NOT false is true.

Wire gates together, with the output of one feeding the input of the next, and you can compute anything a computer can. A couple of gates can add two digits of binary (numbers written using only 0s and 1s). A few thousand gates flew astronauts to the Moon. The chip in your phone has billions.

But you can't train a circuit of gates. The only things you can change are choices like "is this gate an AND or an OR?", and you can't make a gate 1% more like an OR. Its outputs snap between 0 and 1, so a small change either does nothing or flips the answer completely. There's no slope to follow.

Step 1: Values Between True and False

Logic gates work on 0 (false) and 1 (true). But what if, rather than just 0 or 1, an input could be any number between 0 and 1? Then 0.2 could mean "probably not", 0.5 "not sure", and 0.9 "almost certainly". That's closer to the real world, where we're rarely completely sure of anything.

Here's a way to compute AND that works with numbers like these: add up the two inputs, and if the total is more than 1.5, output 1. Otherwise output 0. When the inputs are exactly 0 or 1, only 1 + 1 gets over 1.5, so this is exactly AND.

OR works the same way with a lower bar: output 1 if the total is more than 0.5. Any single 1 gets over that.

NOT needs one more idea: multiply the input by −1 before adding. Then output 1 if the total is more than −0.5. An input of 0 gives a total of 0, which is more than −0.5, so the output is 1. An input of 1 gives −1, which isn't, so the output is 0.

Step 2: The Gate That Does Everything

All three gates follow the same pattern, so we can make one machine that can be any of them:

1. Multiply each input by a number called its weight. The weight says how much that input counts, and a negative weight means the input counts against the answer.
2. Add up the weighted inputs, plus one more number called the bias. The bias replaces the bar. Instead of asking "is the total more than 1.5?", we add a bias of −1.5 and ask "is the total more than 0?" It's the same question.
3. If the total is more than 0, output 1. Otherwise output 0. We'll call this the cutoff.

This is real progress. Which gate we get is no longer a choice between separate parts. It's set by numbers, the weights and the bias, and those are knobs, just like the knobs of the models in Chapter 1. (This add-then-compare machine is called a perceptron, and it was invented in 1958.)

But it's still not smooth. Try nudging the bias in the playground above: the output doesn't change at all, until suddenly it flips from 0 to 1. That's the stepped curve from Chapter 2 again.

Step 3: Replace the Cutoff With a Slope

The fix is to replace the sudden cutoff with a smooth S-shaped curve called the sigmoid. It squashes any number into the range 0 to 1. Large positive totals become close to 1. Large negative totals become close to 0. Totals near 0 get a value in between. Think of it as a dimmer switch instead of a light switch: instead of snapping between off and on, it slides smoothly between them.

The key property for us: when you change a weight by a small amount, the output now changes by a small amount. No sudden jumps. That's exactly what Chapter 2 told us we need.

The Artificial Neuron

Put Steps 2 and 3 together and you have an artificial neuron, the building block of every neural network. A neuron does two things:

1. Weighted sum. Multiply each input by its weight, add them up, and add the bias.
2. Activation function. Pass the total through a smooth function, such as the sigmoid, which decides how strongly the neuron responds.

The name comes from brain cells, which we'll compare at the end of this chapter. Below is the same machine you played with in Step 2, with the cutoff swapped for the sigmoid.

This neuron has two inputs, so that it can work like a logic gate. In practice, neurons can have many more. Neurons in modern AI systems often have thousands of inputs, each with its own weight. The recipe stays the same: multiply each input by its weight, add them all up along with the bias, and pass the total through the activation function.

Three Neurons Solve XOR

Logic gates become powerful when you wire them together, and the same is true of neurons. The simplest example is XOR, which means "one or the other, but not both." A single neuron can't compute XOR: whatever weights you give it, it can't get all four input combinations right at once. But two neurons feeding into a third can. The two middle neurons are called hidden neurons, because they don't produce the final answer. They work out partial answers that the output neuron needs:

• Hidden Neuron 1 computes OR: "is at least one input on?"
• Hidden Neuron 2 computes NAND (not-and): "are they not both on?"
• Output neuron computes AND of those two results: "at least one is on, AND they're not both on." That's XOR.

Neural Networks

A neural network is just many neurons wired together like this, arranged in layers. Each neuron takes its inputs from the layer before and passes its output to the layer after. Every connection has its own weight, and every neuron has its own bias. Together, those weights and biases are the network's knobs, which training adjusts.

XOR needed a single hidden layer working out partial answers. Harder problems need many. You can't jump straight from raw pixels to "that's a cat," so you build up through stepping stones. The first hidden layer might detect simple patterns: edges, color patches, textures. The next combines those into parts: eyes, ears, fur. A later layer combines those into whole objects: "cat" or "dog." GPT-3 has 96 layers, and the most advanced models today likely have more.

No human decides what each layer computes. The network learns that as part of its training, using the same recipe as always: make a tiny change to the knobs that reduces the error. Every weight affects the final output, so for every weight we can work out which way to nudge it to reduce the error (its gradient, from Chapter 2). An algorithm called backpropagation works this out for every weight at once, by flowing the error signal backward through the layers.

A Smooth Universal Approximator

Now we can see why a neural network has both properties from the start of this chapter. Each neuron can act as a logic gate, and logic gates can build any function, so a big enough network can too. And every weight is a smooth knob, so optimization can find the right settings. Smooth logic gates are only one way to think about neural networks, though: trained networks rarely look like tidy circuits, and later chapters offer more realistic pictures, starting with Chapter 4's view of a neuron as a detector.

How Do Networks Compare to Brains?

Artificial neurons were inspired by the neurons in our brains, which also add up weighted signals from many others and fire when the total is high enough. Each connection between brain neurons, called a synapse, plays roughly the role of a weight. So here's how brains and AI systems compare, counting synapses and weights:

There's something strange in this chart. Today's largest language models have roughly as many parameters as a mouse brain has synapses (though comparing artificial parameters to biological synapses is not an exact match). Yet they can write essays and translate languages, and mice generally can't. And elephants have more brain neurons than humans, yet aren't smarter than us. The number of connections isn't everything. What matters most is how good the weights are. That's what training is all about, and we'll get deeper into it in later chapters.

What's Next

A neuron is just a weighted sum and an activation function. But what is that weighted sum, really? In Chapter 4, we'll meet vectors, lists of numbers that describe things, and discover that a neuron's weighted sum is actually a geometric operation called a dot product. It measures how closely the input matches what the neuron is looking for, which turns out to be a powerful way to think about what each neuron does.