Last week, an Anthropic researcher warned that rapidly advancing AI could lead to human extinction in the not-too-distant future. Then tech bosses joined the call for countries and companies to work together to slow the race towards artificial super intelligence. Today, AI philosopher Nick Bostrom on just how bad unbridled AI could get and whether we need to put the brakes on before it’s too late.Featured:Nick Bostrom, AI philosopher Subscribe to ABC News Daily on the ABC listen app.Program:More from ABC News DailyAustralia, AI, Science and Technology, Technology, ScienceTranscriptSam Hawley : We're being warned that if we don't slow it down, AI could kill us all within the decade. But how exactly would it do that? Today, philosopher Nick Bostrom on what happens when AI is smarter than humans. I'm Sam Hawley on Gadigal land in Sydney. This is ABC News Daily.Jacob Coxon, Ex-Anthropic Researcher : It's not just me. Many people working in the industry, including people, executives, senior researchers, all genuinely believe that there is a substantial probability that this technology could kill everyone.Sam Hawley : Nick, you've done a lot of work on the threats that AI brings with it. So when the former anthropic researcher Jacob Coxon declared last week that AI could kill us within the decade, I suppose you weren't so surprised as the rest of us might have been.Nick Bostrom: No, I mean, there's been a lot of talk about this amongst people in the inside and people interested in AI safety and some previous resignations. Daniel Kokotajlo was another AI researcher who resigned.Daniel Kokotajlo, Former OpenAI Employee : We are just not ready for what's about to happen. We're not ready for superintelligence.Nick Bostrom: And I think the impression now is the progress is so rapid that these worries that previously were sort of hypothetical and futuristic are now becoming real.Sam Hawley : And of course, this news, it spread like wildfire. And, well, you might have thought that AI companies would have been scrambling to allay people's concerns. But no, the head of Anthropic, Dario Amodei just days later, he called for the global slowdown of AI.Dario Amodei, Anthropic CEO: It's funny, I agree with Jacob much more than I disagree with him. He wasn't calling out us. He was calling out the dynamic of the industry as a whole moving too fast.Nick Bostrom: Yeah, I mean, I think the level of worry and concern is probably higher inside some of these AI labs than it is amongst the general public and policymakers.Sam Hawley : Well, Dario Amodei, he did speak to CNN. Of course, he penned this almost 4,000-word essay. Now, this came out after the warning from Jacob Coxon.Dario Amodei, Anthropic CEO: If we take the right paths, then the chance of something going wrong is very low. But at the end of the day, the industry has to work together and it's harder. But to some extent, the world has to work together.Sam Hawley : What did you make of that essay?Nick Bostrom: Yeah, I think it's quite thoughtful. And so what he's advocating for is not stopping AI, but having the ability to slightly moderate the pace. To give more opportunity for doing careful evaluation, alignment, safety testing. Right now, it's very difficult for one of these frontier labs, even if they do have safety concerns, if they slow down, even for a couple of months, because they feel their system is not quite ready yet, they just immediately fall behind to a competitor who might take more risks. And they're all in that competitive situation. So in order to allow these frontier labs to spend a little bit more time on doing things right, it might be necessary for all of them to agree to slow down slightly together. But one of them can't do it alone, or they just drop out of the race and become irrelevant.Sam Hawley : And of course, Sam Altman, the head of OpenAI, and Elon Musk, they did come out to agree with him that AI is getting too fast at developing the next generation of AI. That's what it's about, isn't it?Nick Bostrom: Yes, we're entering this phase of recursive self-improvement. So this is when the AI systems are not smart enough to be able to do a lot of the tasks that traditionally human AI researchers are doing. We've seen it for the last couple of years with coding, where AI's are already really good. And increasingly, they can also run various experiments. And as they become smarter and smarter, at some point, the majority of the progress will be driven by these digital minds themselves, right? And then things could really pick up the pace.Sam Hawley : Oh my gosh, yes, because we're talking about artificial super intelligence here, aren't we? So it's not what most of us use, like the chatbots. It goes well beyond that.Nick Bostrom: It goes well beyond that. So obviously, the AI inside the labs is a bit further ahead than the publicly deployed models. But I think these concerns are mainly not driven by the current AI, but what we can see coming down the pike. So if you extrapolate this rapid rate of progress that we've seen over the last two years, forward another six months, 12 months, or beyond that, then at some point, they have the capability where, if they were misaligned, they could potentially cause massive destruction. And eventually even jeopardise humanity's survival.Sam Hawley : Well, Donald Trump, he has brushed off this idea of AI leading to human extinction. He, it seems, Nick, is far more worried that the US will lose the AI race.Donald Trump, US President: All I say is that we're leading China in AI. We're the most sophisticated country in the world. And frankly, I want to keep it that way. Because whoever wins, AI wins. And we can put guardrails, we can do this and that. But I think you have a lot of negative forces in bringing it up.Nick Bostrom: Well, there is competition at various levels. I mentioned earlier between companies, right? But there is also the geopolitical level where the US is in a sort of race with China. And so that's another place at which it is difficult unilaterally to pursue the optimal policy because without coordination, you then risk just falling behind your competitor.Sam Hawley : Nick, let's have a look more deeply at this warning from Jacob Coxon that he issued last week, that in fact, AI could really kill us all. Now, a lot of people since that warning have been asking, well, how? Now, bioweapons is a big concern, isn't it? Just tell me about that. That's one way that could happen.Nick Bostrom: Yeah so that's one way. And that could happen, I mean, in one scenario, maybe if the AI's themselves go completely off the rail. Another way that could happen is if humans misuse these AI agents to assist with designing new pathogens and so forth. And you might particularly worry about the open weights models. So it's one thing if you have, you know, ChatGPT or Claude, OpenAI and Anthropic, they can sort of impose restrictions in what they will help a user with, right? So right now, if you try to get Claude to give you advice on how to make a bird flu more infectious among humans, it will just refuse. But there are also these open weight models, both US ones and Chinese ones and made by other countries. Even if they are trained to refuse these kinds of requests when they're released, it's very easy if you have access to the weights to sort of remove those inhibitions. So it's not clear if you have these open weight models, how you could ensure that people wouldn't use them for these kinds of malicious purposes.Sam Hawley : Wow. So artificial intelligence, it could design new viruses, deadly viruses.Nick Bostrom: Yeah, well, that's one thing. To guard against that, I think we really should be shoring up our physical defences, in particular, maybe by regulating DNA synthesis machines. So that even if you manage to get a blueprint for some nasty thing, maybe using one of these open weight models, there would still be another check when you try to then actually print this out to sort of make a viral genome. You need at some point to combine chemicals to actually get the physical product. And that would be a natural place, I think, to sort of increase scrutiny. But I think the main worry, though, is that eventually these AI systems themselves will become very powerful. And potentially that could lead, once they're super intelligent, to scenarios of human disempowerment, where they are able to gradually seize control over essential processes in the economy. Eventually, robotic persuasion abilities might become superhuman. Once you have a super intelligence that's way beyond the human level, if it is misaligned, it seems just plausible that eventually, if it wants to achieve some goal, it would be able to do that and sort of route around human countermeasures. It's kind of almost like a human versus ape scenario. Even if the gorillas are physically stronger, we have more tanks and stuff, but in the end, human ingenuity will win. You don't want to end up in a situation where you have an adversarial entity that is radically more intellectually powerful than even teams of the best humans.Sam Hawley : Because we could become the ape in this scenario, I suppose. There's also other threats, of course, isn't there, Nick? Like people mentioned the nuclear threat. There's some suggestion of a global drone war led by AI. It's either fanciful or it's really scary.Nick Bostrom: Yeah, I mean, I think the key is to distinguish between what the AI is currently capable of and what we can see that they will become capable of in a few years, especially if recursive self-improvement really kicks in and you get to these radical levels of super intelligence, where science and technology and strategising is happening at digital timescales rather than biological timescales. Once you have AI that can do invention and entrepreneurship and strategising and science at superhuman speed, it's really an across-the-board acceleration. And in that, there is intrinsically just a great amount of power that we need to make sure is deployed for beneficial purposes.Sam Hawley : Gosh, any other ways then that it could actually kill us all, AI? What else could it do?Nick Bostrom: Well, I mean, so in the short run, it might be by manipulating humans. In the longer run, by developing robotic infrastructure that will then dispense with the need of using human arms and legs. Maybe at an even more advanced stage, it could be nanotechnology-based infrastructure. It's easier to see what the end result of that would be than to see the exact path. It's like if you were playing chess against Magnus Carlsen, right? It's easy to predict how the game will end, like Magnus Carlsen will win. It's harder to predict the precise sequence of move that he will play to win, because in order to do that, you would have to be as good at chess as Magnus Carlsen is.Sam Hawley : Well, Nick, in his essay, The Head of Anthropic, Dario Amodei, he also noted that AI could bring a lot of good as well, of course. He says, including curing most major diseases in the next five to ten years. And we've heard, of course, it could find a cure for cancer. Yes. Do you think that's all possible?Nick Bostrom: Yes. So this is what is special with AI. So there's things that are just bad, like nuclear war, nobody's in favour. It's just bad. We just need to reduce the risk and slow it down and prevent it. AI has also this extreme upside, if we get it right. It really could help do so much to improve human flourishing. I'm particularly excited about medical applications. So there is a kind of an urgency on that side as well. Like if there's going to be a cancer cure, it matters a great deal, whether it happens in eight years from now or 10 years or 12 years. There will be a lot of people dying in the meantime if it gets delayed longer than necessary. And then you can apply that in other areas, whether it's for clean energy or for science, for economic abundance. It really could be this massive unlock. So I'm ultimately really excited about the potential of AI. But I do think it's the kind of thing, you know, we might only get one shot at it. So we really need to put our best, very best foot forward.Sam Hawley : Yeah. Alright. So there will be some good, but Dario Amodei and others, of course, they just want the pace to slow down. Now he suggested a three-step plan. One, he wants all AI companies to have a third-party evaluator. Second, that democratic countries come up with safety standards across the board. And third, that the world, the democratic world, tries to pressure authoritarian nations to do the same. Would any of those grand statements really change anything in your view?Nick Bostrom: Yeah, potentially if done right, it would give this option to go a little bit slower and to have standards of safety so that it's not immediately release whatever we can as soon as it reaches some level of capability. But if we need to take a little bit longer to make sure it meets various safety standards, we would have the option of doing that. And at the moment, there is a lot of uncertainty exactly whether or how much of a slowdown would be needed. Because we haven't solved this alignment problem completely yet. We don't know how hard it will be, but it's plausible that it will be quite hard and that then having the ability to take a little bit of extra time could be super valuable. And then on top of that, later on, one could then potentially, if need be, build these regulatory regimes or potentially international cooperation if that proves feasible.Sam Hawley : Alright. So what do you think then, Nick? If we don't slow this whole process down, do you agree with Coxon's prediction that AI will kill us all within the decade?Nick Bostrom: I think it's like there's a lot of uncertainty about this. We might just get lucky and it turns out this alignment problem isn't so hard and you just sort of need to move forward and things will fall into place. That could be the case. Hopefully it is. But given the magnitude of the stakes, we really want to sort of stack the odds as heavily in our favour as we can. And so having a lot of smart people working really hard and being really careful, I think would reduce the doom probability.Sam Hawley: Nick Bostrom is a philosopher. He's been studying the risks of AI since the 1990s. This episode was produced by Ilaria Brophy and Louise Miolin. Audio production by Sam Dunn. Our supervising producer is Sydney Pead. I'm Sam Hawley. Thanks for listening.