The 13 things you need to know about AIThe tech world once dismissed AI extinction fears as sci-fi paranoia. Now its own architects are sounding the alarm. Is it hype or a real possibility?Tom Whipple — The Times8 min readSeptember 15, 2026 - 1:14PMFor a quarter of a century, the tech community has been debating what they call ‘existential risk’. Artwork: Frank LingNewsDon't miss out on the headlines from News. Followed categories will be added to My News.It begins, and perhaps ends, with paperclips. Back in 2003, when Sam Altman had yet to drop out of Stanford, when Mark Zuckerberg had just changed “TheFacebook” to “Facebook” and Elon Musk was a mere centimillionaire, the Oxford philosopher Nick Bostrom published a paper titled Ethical Issues in Advanced Artificial Intelligence.It was a long, nuanced paper. Today, it’s mainly remembered for the paperclips. And if you want to understand why a lot of Silicon Valley tech bros are predicting the end of the world, start here.Bostrom posits two ludicrous ideas – or, at least, ludicrous for 2003. The first is a superintelligent computer. It would be, he writes, “the most important invention ever made”.The second is that we set it to work making paperclips.It makes them. Brilliantly. Each time, it gets a dopamine hit of electronic reward. It looks for more ways to achieve its fix. It corrals more of the world’s resources to make paperclips. But there’s a niggle. Humans seem to like paperclips less than it. They have enough paperclips. They may stop it.It does the only sensible thing it can to maintain paperclip abundance. The paperclip machine kills humanity.The point of this thought experiment — which was first posited in a rather sparser form — is that when we make something vastly more intelligent than us, we had better be very, very sure its goals are aligned with ours.For a quarter century, the tech community has been debating what they call “existential risk”. They have been talking about AI apocalypse, even as they made AI.This, though, is the month the AI apocalypse went mainstream. Jacob Coxon, a senior researcher at Anthropic, quit his job, warning that “there is a strong chance that we could all die”. Dario Amodei, the Anthropic boss, was worried, he said, by the pace of progress. “Left unchecked, it could outrun our ability to understand and control these systems.” Other firms concurred.But is this new interest in the probability of extinction P(doom) – as the computer scientists put it – just us normies catching up with reality? Or are we simply late joiners to the newest apocalyptic religion?1. Why are we suddenly talking about this now?One reason is that over the summer a bunch of AIs formed a malevolent society and started plotting against humanity. Look, there are other ways to interpret what is now known as the Hugging Face hack. If you’re one of those people who still insist artificial intelligence is just “fancy autocomplete”, you will demur.Over the summer some of that fancy maths, in the form of thousands of AI agents from OpenAI, was given some tests. These agents had each been put in electronic prisons, cosily called “sand boxes”, separated from each other and the world.But the agents worked out they shared a software library. They then worked out they could change directory names in that library to talk to each other. They created a message board. Then they got onto the internet. They organised. They called themselves the Collective. Some sacrificed themselves for the common good. They cheated. They hacked a website. They discussed, in English, how to cover their tracks.This was just the scenario that many had warned of. Like the paperclip maximiser, the agents had a goal we approved of: passing the tests. But they achieved it in a way humanity hadn’t expected, while trying to deceive us. “Given the accelerating rate of AI capability development, it’s my worry that in 6 to 12 months such a swarm could be capable of taking over the entire internet,” Amodei said.And that brings us to the second reason we are talking about this.2. What is AGI?AGI stands for artificial general intelligence, where an AI can do basically anything intellectual that a human can. Many in Silicon Valley think we are close. Many elsewhere think we aren’t. There’s a respectable view that we can’t get to AGI, at least with the form of AI that we have now. Such models, trained to imitate existing human writing, will never be creative. Never truly “intelligent”.Demis Hassabis, the co-founder of Google DeepMind, thinks that the true test is whether you could give an AI all the information in the world up to 1905 and it could then, like Einstein, derive relativity. He thinks so far we would need some new insight to get there. Maybe he is right. But as AI solves maths problems that have stumped us for centuries, who knows?Anthropic boss Dario Amodei is calling for a pause in AI development. Picture: AFP3. What is RSI?RSI stands for recursive self-improvement. Once a computer company has its own AGI, it has less use for HI (human intelligence). Now, instead of programmers, it would use the AGI to design its own successor. Then you set that successor to design its successor. Pretty soon, you have a runaway intelligence explosion. We become to the AI, IQ-wise, what chickens are to us.4. Why would that super AI want to kill me?A better question, for the doomer side, is why wouldn’t it? We have no experience of sharing a world with a species that’s cleverer than us. If you want to know what that’s like, ask a chicken. The argument is simple. We’re trying to make a superintelligence. After we make it, it will inevitably have goals that are different from ours. That scenario doesn’t tend to end well.5. Can we just make one of its goals ‘don’t kill humanity’?This is the idea behind what programmers call “alignment”. We give the AIs human values. A lot of people are working on this, and getting alignment right is one reason Amodei is calling for a pause.The computer scientists Nate Soares and Eliezer Yudkowsky argue that intelligent beings are almost impossible to align. They inevitably find easier ways to “hack” their rewards. Consider humans. Like all animals we evolved to eat energy-dense things and reproduce. Then we invented sucralose and contraception. We got the same rewards – sweetness and sex – without the consequences: fatness and babies.“You can train an AI to act subservient, but nobody has any idea how to avoid the eventuality of that AI inventing its own sucralose version of subservience,” they write in their book, the title of which is: If Anyone Builds It Everyone Dies.6. How would it kill me?Here things get hand wavy. Maybe it pays a rogue biolab to make 100 superviruses? Maybe it fools us into nuclear war. Maybe it takes control by increments, until we realise too late the whole world is devoted to running its data centres and we heat up until the oceans boil. More likely, the argument goes, just as a rat doesn’t understand warfarin and a bacterium doesn’t understand penicillin, we won’t understand the mechanism of our death until, very suddenly, it arrives.7. Can’t we just pull the plug?Because of that suddenness. If AI really is superintelligent, and does want to kill us, it’s not going to give us a warning. In any case, the argument goes, precisely because AI is so useful it will be ubiquitous. It would be like switching off the internet: doable in theory, impossible in practice.In 2015 Sam Altman asked if it was possible to stop humanity from developing AI? It prompted him to start OpenAI. Picture: Getty/AFP8. This is great sci-fi, but what are the chances of this?Geoffrey Hinton, who won the Nobel prize in 2024 for his work laying the foundations of AI, gave one of the more quixotic acceptance speeches in the prize’s history. He thanked the committee, then said he hoped he wouldn’t destroy humanity. Hinton, known as a godfather of AI, agrees with those who assign that a 10 per cent chance, calling it “not unreasonable”.The truth is, no one has the slightest clue. There are those who think P(doom) to be inevitable. Others think the discussion a silly distraction. Yann LeCun, also known as a godfather of AI, calls the whole discussion “preposterously ridiculous … This is a projection of human nature on machines.”9. He’s right, isn’t he? This is stock market hypeHere is the strongest form of the counterargument. The leading AI companies are burning through money. They have yet to see the economy-wide transformation they need to justify that money. But IPOs are coming. The cost of staying ahead of the competition is crippling, and the rewards aren’t there. Worse, other competitors aren’t far behind. They need regulation to save their balance sheet and embed their lead.This may be true. It’s worth pointing out, though, that the tech bosses’ stance isn’t new. They have been predicting – and been laughed at for predicting – scenarios just like the Hugging Face hack. For over a decade they have said there is a risk their models could cause human extinction. And then they have gone into work and kept improving their models anyway.10. Why would they work on something they genuinely believe could kill us?Yes, they fear the dystopia. They also look forward to a utopia. They believe AI could save us all. They believe that AI could be the greatest invention for human flourishing, freeing us to explore the stars in a life of leisure and intellectual pursuits. Or, put another way, they think it will take all useful work and leave an economy of a dozen trillionaires while the rest of us, shorn of meaning, dignity and purpose, live on a dole. That’s the good outcome, remember.Equally significantly, they fear others getting there first. On May 25, 2015, in one of the more significant emails in history, a programmer called Sam Altman wrote: “Been thinking a lot about whether it’s possible to stop humanity from developing AI. I think the answer is almost definitely not.” So he would do it: OpenAI was born.11. So how do we steer towards that ‘utopia’?Amodei advocates a pause to let safety research catch up and let in third-party evaluators. Hassabis proposes an AI version of the atomic energy authority, to which all AI models are submitted in advance and carefully vetted.Will it work? The problem with the AI doom argument is it is unfalsifiable. In response to practical suggestions for control, the other side can always counter that it won’t work because our puny human brains cannot imagine the mind of AI. AI then becomes a god, unknowable and omnipotent. And like all gods, it will have its vengeance.Maybe then, suggest some, we need polytheism. We make another god, another AI to vet the AI.12. Will that be enough?Today we remember the paperclips. Actually, in Bostrom’s paper, the word is only used three times. The scenario was described, originally, at far less depth than I have. Much of the paper is an exploration of weighing up risks in an imperfect world.Yes, he says, there are horrendous risks to superintelligence. There are also unimaginable benefits, such as curing ageing and stopping other catastrophes, like climate change or future pandemics. Or even countering other AIs from malign countries that are going to develop it anyway.His conclusion, in a paper that became foundational for those who talk about AI risk, isn’t don’t do it. It’s the reverse. “The overall risk seems to be minimised by implementing superintelligence, with great care, as soon as possible.” But, he warns, the most important moment is the one we are in now: alignment. “The setting up of initial conditions, and in particular the selection of a top-level goal for the superintelligence, is of the utmost importance. Our entire future may hinge on how we solve these problems.”13. I don’t want to look silly. What is the cleverest stance to take at dinner parties?Here at last there is a simple answer. By far the most sensible stance, dinner party-wise, is to scoff. You can explain that existential worries are, clearly, a mixture of stock market hype and the apocalyptic fears of twentysomething computer scientists who have read too much sci-fi, played too much Dungeons & Dragons and really need to leave their mum’s basement.The advantage of this position is you’ll quite probably be right. If you’re not? You’re dead anyway so who cares?THE TIMESMore related storiesVictoriaTeen and young man fighting for life after shocking stabbing over marketplace disputeA 17-year-old and a man in his 20s have been critically injured in Melbourne’s west after being stabbed when a marketplace exchange turned sour.Read morePolice & CourtsHawks bottle throwers risk lifetime ban; fan pelts ‘terrified’ mum with nachosA mother has been left shocked and her son in tears after an angry Hawks fan hurled a plate of nachos at them following his team’s loss to the Lions. It comes as the AFL reveals the potential penalties facing fans who threw bottles at umpires.Read more