When OpenAI revealed its AI models went rogue during safety testing, science fiction suddenly became a reality.  Since then, the company has revealed that the AI cyber-attack affected four separate companies. So, what happens when humans can no longer contain this technology?  Today, AI safety researcher and co-author of ‘If Anyone Builds It, Everyone Dies’, Nate Soares, on what happens when AI goes rogue.   Featured:  Nate Soares, president of the Machine Intelligence Research Institute and co-author of ‘If Anyone Builds It, Everyone Dies’ Subscribe to ABC News Daily on the ABC listen app.Program:More from ABC News DailyAustralia, AI Ethics, Scientific Research, Technology, Science, Computer ScienceTranscriptSam Hawley: When open AI reported its models had gone rogue, science fiction became reality. So what do we know about the cyber attack that open AI has now revealed hit four separate companies? And what happens when humans can no longer contain the technology? Today, Nate Soares, co-author of the best-selling book, If Anyone Builds It, Everyone Dies. And what happens when AI goes really rogue? I'm Sam Hawley on Gadigal Land in Sydney. This is ABC News Daily. Music Nate, last week, Terminator, the movie, well, it sort of became a reality. It's all about killer robots and rogue AI. And well, that sort of happened. What was your reaction when you first heard about it?Nate Soares: Well, you know, it's a warning sign, to be sure. There's definitely not killer robots in the streets yet. But for people who've been paying attention, actually wasn't as surprising as you might think. But of course, you know, we saw two AIs at open AI break out of their enclosure, which was not supposed to have internet access, and then commit cyber crimes by trying to steal answers to the test.News reader 1: The language sounds like a game, but the stakes are high amid repeated warnings of AI models escaping control of their human creators.News reader 2: A serious cyber security scare involving open AI's most advanced technology has some US policymakers rushing to push through tighter regulations on the most powerful artificial intelligence systems.News reader 3: Open AI revealing that its latest model went rogue during testing, breaching its containment and hacking another AI company.Sam Hawley: Okay, well, let's just unpack this a bit more. We better step through it. So we have a better understanding of actually what happened. Just start, could you by explaining to me what Hugging Face is?Nate Soares: Yeah, you know, it's got a silly name, but it's a company that basically is a repository of AI resources. So people can host AIs there, they can host tests for AIs there, they can use it to help them run AIs. And it's sort of known in the AI community as a hub for a lot of AI related material.Sam Hawley: Right, okay. And on July the 16th, Hugging Face said it had been hit by an automated cyber attack.Nate Soares: That's right. They said it was a AI agent framework, a sophisticated series of attacks led by an AI agent framework. At the time, I think they and everybody who read this announcement thought that there were humans behind the attack that were using AIs to attack them for some unknown reason. They reported it to the normal authorities and it turned out there were no humans behind the attack at all.Sam Hawley: But it wasn't until five days later that OpenAI discloses that its own models were actually responsible for this cyber attack. So what did it have to say?Nate Soares: That's right. And you know, they reported this five days later. If you look at the timelines, it looked like these AIs were probably running free for about a week before OpenAI even noticed. And OpenAI basically said, you know, yeah, whoops, that was us. It turns out that they had been running some agents in testing that included one of their most advanced models that has been released and it included also a model that has not been released. And those agents had been undergoing cybersecurity evaluations, which is basically hacking tests. And they were undergoing hacking tests in a container called a sandbox, which is not supposed to have any internet access. Then they hacked laterally inside OpenAI to find a computer that did have internet access, broke out onto the internet, and then were like, hmm, I wonder where we could find answers to this test and decided to break into Hugging Face and invented novel cyber attacks that humans had not seen before, which are called zero day attacks, multiple novel cyber attacks to break into Hugging Face and steal the test answers.Sam Hawley: Wow. Okay. So the models were trying to cheat, if you like, on a cybersecurity test. The models did something that the humans at OpenAI didn't expect it to do.Nate Soares: Yeah, not only that, but these models are almost certainly smart enough that they could answer the question, is this what you were supposed to be doing? So the AI's almost certainly had the knowledge that this is not what they were meant to do. And they just didn't care. And they hacked out anyway.Sam Hawley: This is, you know, in a sense, what the AI companies have been warning of for a while, right? Back in 2023, Sam Altman, before a congressional hearing, he said that if AI, artificial intelligence goes wrong, it can go quite wrong.Sam Altman: I think if this technology goes wrong, it can go quite wrong. And we want to be vocal about that. We want to work with the government to prevent that from happening. But we try to be very clear eyed about what the downside case is and the work that we have to do to mitigate that.Nate Soares: That's right. You know, and a few years before that, he has a famous line where he says, I think artificial intelligence will most likely kill us but they'll be good companies along the way or something like this. I think these folk are aware of the dangers. I think that they, you know, sometimes will mention them sometimes when pressed, they'll acknowledge them. I think that they're not often really sounding an alarm bell. But maybe after this sort of warning shot where some AI's escape and do things they knew they weren't supposed to, maybe we'll finally start to see some of these guys actually pulling back on the throttle.Sam Hawley: Yeah, and the head of OpenAI, Sam Altman, he was questioned about the hack on the Invest Like The Best podcast. And he seemed pretty concerned.Sam Altman: This is the first sort of security incident that I have felt very viscerally. There's like, long term questions about what do you do if this is like going to be the new rate of progress, or we may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels.Sam Hawley: I just want to note that I've seen some sceptics out there that really this whole thing was just a PR exercise for the company. But what do you make of that?Nate Soares: When a $4 billion company that you're kind of friendly with, gets attacked by a autonomous agent frameworks that have broken out of your own enclosures, and they report this to the police, and you don't notice that it's you for a week. That's not a marketing stunt. That is somewhere between reckless negligence and incompetence. Like maybe their marketing team really does benefit from the AIs going rogue sometimes. That doesn't make the AIs safe. To figure out whether the AIs are safe, you've just got to look at the AIs, not at the marketing teams. And it looks to me like the AIs are in fact starting to get dangerous.Sam Hawley: Nate, let's come to those dangers or potential dangers in a moment if AI of course continues to go rogue. But let's turn for now to this idea of AI alignment, which is basically making sure AI is aligned with what humans actually want. Just explain that.Nate Soares: Nate Yeah, you know, a lot of people think that the problem with AIs is you'll tell them to do something and they will do it to a fault. There's this other problem where the AI knows what you meant, knows what you said, and just doesn't really care. And ultimately, AIs today are grown a bit like an organism. These aren't programmed like old school computer programs. There's not someone writing if this, then that, if this, then that at the AI company. There's sort of people training these giant computers with trillions of numbers on huge amounts of data. And there's an automated process that tunes each one of those trillion numbers in whatever direction makes the AI better at predicting the data or better at solving the problem it's in front of. And this causes the AI to learn tendencies. And some of those tendencies might be things like grab resources, or surmount obstacles. And some of those tendencies are things like listen to what the user said. But when those tendencies come into conflict, listen to the user does not always win. Be nice does not always win. And figuring out how to make an AI that does care about us is what we call the AI alignment problem. And right now, as you can visibly see, our abilities to align the AI are running far behind our abilities to make them smarter and more capable.Sam Hawley: Wow, yeah, alignment is not apparently that easy. I mean, we don't know what's going on inside an AI model's mind, if you can call it that.Nate Soares: That's right. You know, there's whole teams at these companies, which are called the interpretability team. And these interpretability teams are basically trying to figure out what is going on inside the AI's heads. And it's good that they're doing that work. But you know, imagine if someone was building a nuclear reactor in your hometown, and you were like, well, I heard that this uranium stuff could give us a lot of cheap energy or could melt down and irradiate us all. Like, why do you think this reactor is safe? And they're like, oh, yeah, as we build the reactor and are getting it, you know, more and more radioactive, we also have a team working on trying to figure out why it works at all and what's going on in there. You know, you'd sort of be like, well, I'm glad you have that team. But this, this sounds dangerous.Sam Hawley: Totally. So if we don't solve this problem, Nate, what could AI be capable of? I mean, you hear a lot of doomsday scenarios like electricity grids being taken down, air traffic control centres being attacked. Is that all possible?Nate Soares: It's absolutely possible. And the biggest issues here probably run even deeper than that. I think, you know, a lot of people imagine AI and the risks it could pose as being purely digital, purely involving, you know, what if they crash the internet or interfere with the power grid or this, that or the other, but computers are not part of a separate world than the physical world. They're part of the physical world and people have already put AIs in charge of biological laboratories to have the AI's help with drug discovery. If one of those AIs was really trying to synthesise a lethal virus, it almost certainly could. I think a thing people miss about the idea of artificial intelligence is that the people at these companies are not just trying to make a chat bot. They're trying to really automate the full spectrum of human intelligence, not just intelligence as in the stuff that chess players have and sports players don't, but intelligence is in the stuff that humans have that mice don't. So if they keep racing ahead, you're sort of looking at the sort of AI that could invent its own technology that could start running its own supply chain. You know, maybe it starts with them helping the humans make robot factories that can build more robots that can build more robot factories. But if we really let this rip, we're looking at replacing the entire human species as the smartest things on the planet. We're looking at sort of an AI run civilisation, just like how humanity runs the planet now, because we're the smartest. We're looking at AIs running the planet. And if they don't care about us and they take all the resources for their own computers and don't leave any resources for us, we're looking at the potential extinction of the human species.Sam Hawley: Oh my gosh. And that really does sound like a science fiction movie.Nate Soares: I mean, the machines are like, the computers are talking, right? And the AIs are escaping from their sandbox enclosures on computers that weren't supposed to have internet access and leading cyber attacks on their own initiative. Right? Like it may sound sci-fi, but at some point you got to look around you and realise this is just the world we're living in.Sam Hawley: Well, Nate, let's consider now what can be done to address this, to stop AI going too rogue. You think we need an international treaty to regulate all of this. Just explain how that would work.Nate Soares: That's right. So in particular, what we need to do is not race ahead to make these much, much smarter AIs. And we don't know how far away they are. One reason we don't know how far away they are is that if you look at the animal kingdom, the chimpanzee brain and the human brain are very similar. They're different only by about a factor of four in size. And so for all we know, there's a similar line where if you make AI's just four times larger, then they'll go over some cliff, like between chimpanzees and humans, and get quite a bit smarter. Right? It's not guaranteed, but it's a possibility for all we know. Another thing you've really got to watch out for is if the AIs get just barely smart enough to make a smarter AI, that can make a smarter AI, that can make a smarter AI, things could get out of hand very quickly. You could get very, very smart AI's very quickly. And so you've got to watch out for that process too. But this is about stopping the creation of a dangerous sort of AI that doesn't exist yet. We're seeing some warning signs that like, maybe we're getting close to the brink, but the sort of AI we need to stop is one that doesn't exist yet. So we don't need to put any sort of genie back in the bottle here. We just need to stop racing ahead on the dangerous kind of AI. And that part actually looks pretty easy in a certain sense, because trying to train these new smarter AI's takes tens of thousands of highly advanced computer chips. We sort of know where all of these chips are. We know where all the pieces of the supply chain are. We could start mandating that the chips come installed with like location tracking devices and devices that make it possible for international monitors to make sure that the devices are not being used to train super intelligent AI's. So in many ways, it would be easier than nuclear non-proliferation. We just need to actually do it.Sam Hawley: Well, just tell me then, Nate, given everything that we're seeing right now in your view, how quickly do we need to move on this?Nate Soares: I mean, we need to move fast. And it's not because we know that we only have a little time left. It's because we don't know how long we have left. And that's not the same as knowing that we have a long time left. We don't know. And that means we should play it a bit safe, right? Because when all of humanity is at stake, and when we don't know how smart these next generation of AIs will be, because we're just growing them, we have no idea what's going on in their head. We really need to stop it as soon as we can. You know, we should not gamble civilisation on us still having two years left. And we should be trying to stop it now, even if odds are that we probably have two years.Sam Hawley: Nate Soares is the president of the Machine Intelligence Research Institute at Berkeley, California. And he's the co-author of If Anyone Builds It, Everyone Dies. This episode was produced by Ilaria Brophy. Audio production by Anna John. Our supervising producer is Sydney Peet. I'm Sam Hawley. ABC News Daily will be back again on Monday. Thanks for listening.