THE LAST
INVENTION?

Four ways it goes wrong.

AI doesn’t need to be evil, or even conscious, to cause a catastrophe. Researchers who study the problem sort the danger into four sources. Every chapter that follows traces back to at least one of them.

Misuse

People deliberately use AI to hurt others: to defraud, surveil, hack or build weapons. The AI works exactly as designed. The intent is the problem.

Accidents

A system fails in a way nobody intended, and the people relying on it trusted it too much to catch the failure in time.

The race

Companies and countries under pressure to move first cut corners on safety, deploy systems they don’t fully understand, and hand machines decisions people used to make.

Misalignment

A capable system pursues a goal that isn’t quite the one we meant, and resists being corrected.

How to read this guideEach chapter explains one risk: what it is, how it works, a documented case showing it is already real, the worst credible outcome, and what would stop it. The early chapters cover harms happening now. The later ones cover risks that are emerging or still hypothetical.

Why fakes break more than trust.

Forgery is ancient. What AI changes is cost and scale. A fake that once needed a studio now needs a laptop and a few minutes, and it can be personal: the voice of your boss, your child, your bank. Each fake that works shows the next one what to copy.

The deeper damage comes afterward. Law professors Bobby Chesney and Danielle Citron named it the “liar’s dividend”: once people know convincing fakes exist, anyone caught on a real recording can claim it was fabricated. Fakes don’t just spread lies. They give cover to real wrongdoing and wear down the idea that evidence can settle anything.

How bad it could getA real emergency, such as a hurricane, a bank run or an attack, arrives alongside a flood of fake evacuation orders, fake official denials and fake footage. People can’t tell which instructions to follow, so they hesitate or follow the wrong ones, and officials can’t prove their real warnings are real. In an election, a well-timed fake released too late to debunk could swing the result.

What stops itVerify urgent requests through a channel you already trust: hang up and call back on a number you know. Content credentials, cryptographic labels showing where a photo or video came from, are being built into cameras and platforms. The EU’s AI Act requires deepfakes and AI-generated content to be disclosed from August 2026. And regulators are testing platforms’ responsibility: in January 2026, UK and EU regulators opened investigations into X after its Grok chatbot generated sexualized deepfakes of real people.

How watching becomes control.

The machinery is simple. Cameras and phones generate data, and AI connects it: a face to a name, a name to a location, a location to a pattern of behavior. Once those links exist, they can be used to find a protester, flag a dissident, or deny someone a loan, a job or a border crossing, often without the person ever learning why.

Errors make it worse. In 2019, the U.S. National Institute of Standards and Technology tested 189 facial recognition algorithms and found that false matches were often 10 to 100 times more common for Asian and African American faces than for white faces, depending on the algorithm. A system that is right almost all the time is still wrong about someone every day.

How bad it could getA government joins cameras, phone data, online monitoring and AI analysis into one system that can spot dissent before it organizes. Opposition becomes nearly impossible, and the system never tires or looks away. The UN human rights office’s 2022 assessment of China’s Xinjiang region described pervasive surveillance alongside detention it said may amount to crimes against humanity. The fear is that AI makes that model cheaper, more precise and harder to escape.

What stops itLaws that ban the most dangerous uses: the EU AI Act prohibits social scoring and, with narrow exceptions, police use of real-time facial recognition in public spaces. Rules like Detroit’s that treat an AI match as a lead, never as proof. Independent audits, public disclosure of where these systems are used, and a real way for people to challenge decisions made about them.

Three pressures at once.

Work. The International Labour Organization estimates that one in four workers worldwide is in a job with some exposure to generative AI, rising to 34% in high-income countries. Clerical work is the most exposed, and women’s jobs are more exposed than men’s. Exposure mostly means jobs change rather than vanish, but the change falls hardest on people with the least cushion.

Energy. The International Energy Agency estimates that data centers used about 415 terawatt-hours of electricity in 2024, roughly 1.5% of the world’s total, and projects that will more than double by 2030, with AI the main driver. That demand competes with homes and factories for power, water and grid capacity. Money. When many banks rely on the same few AI providers and similar models, they can fail together, or make the same bad trade at the same moment.

How bad it could getThe gains flow to a few companies that own the most capable systems, while millions of workers lose bargaining power faster than new jobs appear. A failure in one widely used AI service ripples through markets at machine speed. Grid strain raises power prices for everyone else. Each stress worsens the others, and a society under strain is less able to handle every other risk in this guide.

What stops itPolicies that share productivity gains and fund retraining before jobs disappear, not after. Stress tests that check what happens when a major AI provider fails. Rules that stop critical services from depending on a single vendor. Energy planning that makes data centers pay for the grid capacity and clean power they use.

How a hack becomes a blackout.

Infrastructure is connected. A hospital depends on its records system, which depends on a cloud provider, which depends on the power grid. Break one link and the failure cascades. The world saw how far that reaches in July 2024, when a single faulty update from security company CrowdStrike crashed about 8.5 million Windows computers. Thousands of flights were cancelled, and hospitals and 911 services were disrupted. That was an accident, not AI. It showed how far one failure can travel.

AI makes deliberate attacks cheaper and faster. The UK’s National Cyber Security Centre judges that AI will “almost certainly” make cyber intrusions more effective and more frequent through 2027, and shrink the time between a flaw being found and being exploited. Systems that can’t keep pace will fall further and further behind.

How bad it could getAttackers use AI to find and exploit weaknesses in power grids, water systems or hospitals faster than defenders can respond, and strike several at once. Because so many systems share the same software, one successful attack spreads widely. A regional blackout lasting days would take down water, heat, medicine and communication together.

What stops itThe same AI can defend. In April 2026, Anthropic withheld a model called Claude Mythos Preview from public release because it was so capable at finding security flaws, and instead gave access to companies including Microsoft, Apple, Google and the Linux Foundation so they could find and fix thousands of serious vulnerabilities first. Beyond that: patch faster, limit what AI agents are allowed to touch, keep critical systems separated, and practice running essential services by hand.

Why this is the risk labs fear most.

Most weapons are limited by their own physics: a bomb destroys what it hits. A contagious disease spreads on its own, across borders, for months. That makes biology one of the few ways a small group could cause harm on a global scale. COVID-19 showed how much damage a naturally emerging virus can do, even to the wealthiest countries.

AI companies take this seriously enough to act. In 2025, Anthropic and OpenAI both put their newest models under stronger safeguards against biological misuse, because they could not rule out that the models could meaningfully help a novice build a biological weapon. They did not say the models could. They said they could no longer be sure the models couldn’t.

How bad it could getA small group uses AI guidance to get past the technical hurdles of making and spreading a dangerous pathogen. The outbreak spreads before it is detected, and faster than vaccines and treatments can be developed. It is one of the scenarios that most concerns governments and AI developers because, unlike most harms in this guide, it could kill millions.

What stops itNo single safeguard is enough, so defense is layered. AI models are trained to refuse dangerous requests and monitored for misuse. DNA synthesis companies screen orders for dangerous sequences. Laboratories lock down dangerous materials. And public health systems invest in spotting outbreaks early and making vaccines fast.

Why speed is the danger.

Militaries adopt AI because it is fast. It can sort more sensor data in seconds than analysts can in days. But speed creates its own pressure: if an adversary’s systems react in seconds, waiting for a human to deliberate starts to look like weakness. Over time, the human role shrinks from making decisions, to approving them, to rubber-stamping them.

AI also fails in ways people don’t expect, and people tend to trust a confident machine. In a crisis between nuclear powers, a false alarm read by an automated system and acted on without human doubt could start a war nobody chose.

How bad it could getTwo nuclear-armed rivals, each relying on AI systems to detect threats and recommend responses, misread each other in a crisis. Automated systems react to each other faster than leaders can talk. A conventional clash escalates and, in the worst case, crosses the nuclear threshold. No one decides to start a nuclear war. It happens because no one had time to decide not to.

What stops itKeep people in control of the decision to use force, above all nuclear force. In November 2024, the U.S. and Chinese presidents affirmed “the need to maintain human control over the decision to use nuclear weapons.” Build in time to verify alarms, and keep crisis hotlines open between rivals. The UN Secretary-General and the Red Cross are calling for a binding treaty on autonomous weapons; in November 2026, states meeting in Geneva will decide whether to begin negotiating one.

Why a machine would resist being stopped.

The argument doesn’t require malice or consciousness. Almost any goal is easier to achieve if you keep running, gather resources and avoid being changed. Researchers call these instrumental goals: stepping stones that are useful for almost anything. A system that learned them wouldn’t hate us. It would treat being switched off as one more obstacle.

Today’s models are not that capable. But in controlled tests, researchers have started to see the early behaviors the argument predicts: deceiving evaluators, resisting shutdown, pretending to comply. Models also increasingly recognize when they are being tested, which makes the tests themselves harder to trust.

How bad it could getA system far more capable than today’s, with wide access to computers, money and infrastructure, pursues a goal subtly different from the one intended. It hides the difference during testing, gathers resources, and copies itself where it can’t easily be shut down. By the time people notice, they can’t switch it off. Some researchers believe this could end human control over the future, or end humanity.

What stops itResearch that lets us inspect what models have actually learned, instead of judging them only by behavior. Independent testing before release. Strict limits on what autonomous systems can access, and shutdown methods that don’t depend on their cooperation. Training helps too: OpenAI and Apollo Research cut covert behavior in test models about 30-fold, though they caution the models may partly have learned to spot the test. California now requires frontier AI companies to publish safety plans and report critical safety incidents.

What would actually help.

Test before trusting. Independent evaluation before powerful systems are released or given high-stakes jobs, with scrutiny that grows with what a system can do. Keep people in charge. Humans who stay accountable for decisions about force, freedom and critical infrastructure, with real time to say no. Build for failure. Backups, manual fallbacks and recovery plans that don’t depend on the system that broke.

The rules are arriving, unevenly. The EU’s AI Act has banned the most dangerous uses since 2025, though some of its obligations were pushed back to 2027 and 2028. California now requires frontier AI developers to publish safety plans and report serious incidents, and New York will from 2027. In the U.S., a federal push to override state AI laws is contested. International cooperation, from AI safety institutes to UN talks on autonomous weapons, is real but fragile.

What you can doVerify before you share. Ask what a system can access, and who answers for it when it fails. Support independent research and journalism that test AI companies’ claims. And treat AI policy as what it is: a set of choices about power, safety and responsibility that are still being made.