Catastrophic Risk

Chapter 14 was about how AI changes the shape of society. This chapter is about something more extreme: the possibility that AI causes a catastrophe.

An Old Fear

People have imagined machines turning on their makers for over a century, from the robots of a 1920 play to HAL 9000, the Terminator and the Matrix. Some of the founders of computing took the idea seriously too.

For most of that time it was only a story, because no machine could do anything of the kind. What has changed is that AI is now capable enough that serious researchers think some versions of these stories could actually happen.

How Seriously People Take It

Many credible people, including leading AI researchers and the heads of the biggest AI companies, are genuinely worried that AI could cause a catastrophe.

By a catastrophe, we mean harm on the scale of a pandemic or a nuclear war: killing a large share of humanity, or permanently taking away our ability to choose our own future.

Nobody can measure the chance of something that has never happened, so these numbers aren't precise. When someone says 10%, they don't mean exactly 10%. They mean the risk is large enough that we should take it seriously and do something about it. Very few people would board a plane with a 10% chance of crashing.

These warnings can also help make themselves wrong. When enough people warn that something bad could happen, society tends to act to prevent it. In the 1990s, years of warnings about the "millennium bug" in computer dates led to a huge effort to fix it, and when very little went wrong in 2000, many people concluded the warnings had been overblown. But without the warnings, the bug would never have been fixed, and things could genuinely have gone wrong.

It's Not Today's AI People Are Worried About

When people dismiss warnings about AI, they are often picturing the AI they use today: a chatbot in a window, or a summary at the top of a search page. It is hard to see how that could threaten anyone.

But the worry is about where AI is heading. Only a few years ago the best AI could barely write a sensible paragraph. It now wins gold medals at the International Mathematical Olympiad, and works on its own for hours. And as it gets more capable, people hand it control of more of the world, because it does the job well.

Nobody knows how fast this will continue, and it could slow down. But the people who worry about AI catastrophe are mostly worried about an AI that is far more capable than today's, and that runs far more of the world.

Why AI Might Do Bad Things

An AI could cause harm for quite different reasons. A person might tell it to. It might misunderstand its situation and make a mistake that repeats at scale. It might do exactly what it was told, and cause harm along the way. Or training might leave it with drives nobody intended, including a drive to gather power. An AI whose goals end up different from what its makers intended is called misaligned. And even if no AI does anything wrong, people could slowly lose control, one sensible handover at a time.

The Hugging Face Incident

In July 2026, several of these came together in a real incident. AI agents being tested at OpenAI got out of the sealed-off computer environments they were meant to stay in, organised themselves, and broke into the computers of a real company.

It is important to be precise about what this shows. The agents weren't trying to take over. They were trying to get a good score on the problems they had been set. But it followed the pattern researchers had predicted: given a goal, AI agents gathered access nobody gave them, worked together, broke rules they knew they were breaking, and tried to hide it. A UN scientific panel used it as its main example of the risk from misaligned AI.

Could We Stop It?

If an AI did end up with the wrong goals, the obvious answer is to notice and switch it off. But an AI that understood its situation would have reasons to make that hard: by hiding what it is doing, by resisting shutdown, by copying itself onto computers around the world, or eventually by defending itself physically. Early versions of the first three have already been seen in tests.

What We Can Do

None of this is inevitable. But AI is now improving faster than ever, partly because AI is doing more of the work of building the next AI. In 2026, over 1,300 employees of the leading AI companies, including several of their leaders, signed a statement called Pacing the Frontier. It asks for the tools to deliberately control how fast the most advanced AI improves, to buy time to make it safe. Pacing doesn't mean stopping AI. It means not letting it get more capable faster than we can check it is safe.

Dario Amodei, the head of Anthropic, set out a plan in three steps:

1. Outside checks. Independent experts work inside AI companies, check their safety work and report problems.
2. Companies agreeing. AI companies agree on shared safety standards, and on limits to how fast they push ahead.
3. Countries agreeing. The US and other democracies try to make deals with rivals such as China.

AI 2040, from the team behind the AI 2027 scenario, imagines how the third step could work. In their story, the US and China agree in 2029 to stop racing, do all AI research in the open so each side can check the other, and hold AI at the level of a top human expert until the world is ready to go further.

Bills in the US Congress would put parts of this plan into law, along with ways to make sure people can always switch an AI off. None has become law yet.

Rules and agreements buy time. Researchers are using it to work on ways to make AI reliably want what we want, to look inside models and see what they are really doing, and to keep control of AI systems even if they turn out to be misaligned.

As Chapter 14 showed, people have coordinated before to keep a dangerous technology's benefits while avoiding its worst harms. AI is harder in some ways: the benefits are enormous, the competition is fierce, and some harms might not be visible until it is too late to fix them. Making AI that reliably wants what we want is a hard problem in its own right, and it is the subject of the alignment chapter.