EVANS, Mr Maximillian, Associate, Global Shield Australia SIMPSON-YOUNG, Mr William (Bill), Chief Executive, Gradient Institute WHITTLE, Mr Devon, Associate, Global Shield Australia [15:06] CHAIR: Welcome. I understand that information on parliamentary privilege and the protection of witnesses giving evidence to this inquiry has been made available to you. Do you have any comments to make on the capacity in which you appear today? Mr Whittle : I'm the Australian director of Global Shield Australia. CHAIR: I now invite you to make some short—emphasis on the short—opening statements. We'll then move to questions from committee members. If there is anything further you'd like to provide in writing to the secretariat, please do so. Mr Simpson-Young : Thanks a lot for inviting me. I'll just say a little bit about Gradient Institute and a little bit about me so you understand the scope of areas I can hopefully help you with. Then I'll say a little bit about the subject of the inquiry. Gradient Institute is an independent Australian not-for-profit research institute set up in 2019. We're a team of AI researchers and engineers who do research on AI safety, ethics and governance and help government and industry put that research into practice. Here are some examples to give you a sense of the scope. We do research into how AI agents interact with each other, the risks that emerge from that and how to control those risks and mitigate them. Our most recent work on this was commissioned by the Department of Industry, Science and Resources and was released by the Australian AI Safety Institute as their first research publication in August this year. We do a lot of work with the government on developing guidance for businesses and we've been doing that since 2019. For example, we developed the guidelines for AI adoption that were released in October last year. This is the key guidance for all Australian businesses—small businesses and large—on how to govern AI. We collaborate with some of Australia's largest companies—banks, insurance, super funds and so on—on how to develop and deploy AI safely and responsibly. We also work with Australian start-ups and scale-ups on how they can build technologies that help AI be deployed safely and securely. We co-founded and co-host the Australian AI Safety Forum, which has so far taken place in 2024 and 2026, with the assistance of the government. It is helping to build a community in Australia of policy people, technical people and product development people who care about AI safety and put that into practice. Quickly, about me, I've been working in AI research and development since the late eighties. I've worked in research and development for global companies based in Australia, deploying AI-related technologies into products globally. I'm on the New South Wales government's AI Review Committee and the ANU School of Computing advisory board, and in 2024 I was on the government's temporary AI expert group, advising the government on AI regulation. Quickly, I'll just make a few comments on the topic. AI is a transformative technology—we all know that—and it's got many benefits, but you can't have those benefits unless you can develop and deploy it safely. I know that's fairly obvious. We do that with aeroplanes, cars, strollers and sunscreen. It's pretty obvious that safety is a foundation for any innovation and any way of getting benefit from technology. The risk profile of AI continues to get more complicated, though, and keeping frontier AI systems safe is becoming harder, as we've all seen. It's particularly the case where agents are interacting with other agents, by design or incidentally, and we've done a lot of work on this in our multi-agent risk work, which I'm happy to talk about. We show that a system of individually safe and reliable agents is not necessarily a safe and reliable system, and this is something a lot of people don't get. People are just talking about an agent doing something; it's much worse when you've got a—yes, you might have some safe agents, but, if you have a group of agents, they can have harms that are not visible for a single agent. The risk from highly capable agents has been obvious in the OpenAI-Hugging Face incident, as we know. But, in that case, which I'm sure you all are familiar with, the agents colluded and planned attacks. They cracked out of a system and gained access to Hugging Face systems across the internet without permission. There were 1,200 agents coordinating that, with 700 of those agents participating in that attack. One of the most surprising aspects of that situation was that the AI agents in the labs weren't told to work together. Each was meant to work alone. They found each other through messaging boards, they coordinated in ways that no human was involved in designing or overseeing, and then they set about causing the harms they caused. So now, with advanced agents, without adequate constraints, wherever agents can leave a mark somewhere on the internet and other agents see that, you've got, potentially, groups of agents colluding in ways that we can't necessarily see. This is just one example. In the submission that we made to this committee, we just talked about three things we were suggesting or recommending. There's substantially increasing the funding to the AI Safety Institute. The risks from AI and AI agents in particular are increasing. Having a technical body inside government that can help keep Australia safe, I think, is critical. It performs a different role from ASD, of course, and I think that funding should be increased. We think we should be introducing mandatory AI incident reporting, which I'm sure my friends here will talk about. But also I think that we should be backing an Australian industry to build AI safety and AI security technologies within Australia and that a specialised investment fund for that that's specifically on safe, secure infrastructure for AI would put Australia in a really good position globally, and I can talk more about that if you're interested. CHAIR: Thank you, Mr Simpson-Young. Mr Whittle : Thank you, Chair, and thank you, committee, for the opportunity to appear before you. Global Shield Australia is an independent, non-profit policy organisation, and we work on reducing Australia's exposure to catastrophic risk. We work across all hazards, but a large portion of our work focuses on AI risk, critical infrastructure security and supply chain resilience. I'll just highlight three points, in this opening statement, from our submission, and then I'm happy to engage further with you all. The first point we'd like to emphasise is that advanced AI poses catastrophic-level risks, including for Australia, whether this is from malicious misuse, such as cyberattacks or biological weapons; whether it's a loss-of-control scenario or misalignment of an AI system; or whether it's from the concentration of economic or political power or the enablement of authoritarian regimes. Second, Australia has agency, so what we do makes a difference, and we can make a difference in responding to these threats. But, to do so, we really need to invest in the areas of advantage that we have in the AI supply chain, whether that's compute, critical minerals or AI assurance technologies. These are areas where Australia could be making a real difference. And we should also be using our regulatory frameworks to encourage a race to the top on AI security and governance. The third point I'd like to emphasise is three specific actions that we'd be advocating for the government to take and for this committee to recommend. We have a range of other ones in our submission as well. The first thing is that Australia needs to harden its defences. As Bill said, the risks are increasing, whether it's from biological weapons, whether it's from cyberattacks or whether it's from the misalignment problem. We need to be ready and prepared to respond when things go wrong, and there are things in that space Australia can be doing. Whether it's regulating the import of synthesised DNA products or reforming our critical infrastructure legislation, there are very concrete actions the government can be taking. Second, Australia should be leading internationally. That includes in forums like the United Nations—it was great to see Australia sign up to that statement from other middle powers at UNGA recently—but also in our region, with our partners in ASEAN and the Pacific. We could be working on common rules around AI. Incident reporting and information sharing are concrete things we could be doing there. Finally, we do need to legislate. We at Global Shield previously identified a range of actions and legislation the government could be putting in place in a report we released in August called Guardrails &gap analyses. The one I really want to focus on today for you is incident monitoring and reporting, which, as you know, came up quite a bit in the committee's hearings yesterday. To contextualise this, every day in Australia, companies are required to monitor when their products and systems go wrong and report this to government. In aviation, for automobiles, for pharmaceuticals, in the consumer safety space—all of these companies are reporting to government when they're causing harm. We think that same system should apply for AI. Thankfully, it does seem that the government is on track to legislate something in this regard if we look at the consultations that are happening around the Australian standards, but it's critical that this system is designed correctly. In particular, we really want to encourage us not to just be copy-pasting what's happening overseas. We know some representatives yesterday were referring to the California model and the New York model. These were good starting points, but we need to make sure this system is fit for purpose for Australia. In this regard, I'd highlight three things. The first is we need to ensure that our system obliges these companies to monitor their products and systems and how they're being used; we can't rely on voluntary monitoring. We also need to make sure that what they are monitoring for and what they are reporting is sufficiently broad so it captures things like the Services Australia incident we saw earlier. Currently, under the California and New York thresholds, that isn't reportable, but we need to make sure that our system would require that to be reported. Second, we need to make sure there are paths for voluntary reporting. Senator Pocock had some good questions to the previous witness around certain sorts of AI harms and how they could be reported. One of the ways we can make sure that these are being reported is by allowing victims of AI harms to make those reports to government for further follow-up. The third thing that we'd want to emphasise is that we need to make sure that whatever agency is getting these reports has the capabilities and the powers to assess them and respond to them. We are currently putting together a fuller proposal for the Office of AI's consultation—that's due on Friday—where we go through this in more detail. We're happy to step through any of those elements further. We think that AI incident monitoring reporting is going to be a critical piece of what Australia needs to put in place to be responding to AI risk. It's going to provide key intelligence to government around risk, enable the government to act to respond to threats and harms and ensure that Australians feel they have an avenue to have their voice expressed when they are impacted by AI. We thank the committee for its important work and would welcome any questions. CHAIR: Excellent. Thank you to you both for providing those opening remarks. I think they go directly to some of the big issues in front of us as a committee. We have been hearing from a variety of different witnesses who are raising similar issues. I'm really keen to get some reflections from you both. You mentioned a couple of, for lack of a better word, low-hanging fruits—some of the things that we can immediately put into place to better regulate and better provide safety and security—but what do you think is necessary for Australia to be able to play a leading role in how we meet some of the significant safety challenges that are present in these frontier models? Mr Whittle : I'm happy to jump in first on that. I think the first thing—and this is what we say in our report—is that we have agency, but Australia needs to invest to develop that agency further. We really need to make sure that we are investing in those parts of the AI supply chain that will give us leverage in particular and add value to our engagement with other partners overseas. Compute, critical minerals, AI assurance—if we can develop strong sectors in these areas, that would make us valuable to our overseas partners, the US in particular, and will help us to ensure that we have a voice in the governance of this technology. More concretely, when we think about what we could be doing domestically, the critical infrastructure security legislation is currently undergoing a reform. Legislation is expected in parliament hopefully this year, or maybe early next year. There are two key aspects of that that contribute to this AI safety debate. The first one is making sure that AI data centres are properly covered by that act. That would require them to manage their risk and report to government when things go wrong. It would also give the government powers to intervene in those data centres when there are incidents. The second thing that act should do, which isn't on the agenda yet, is cover the AI companies themselves. At the moment, something we've been raising with Home Affairs and the government quite a bit is the fact that AI models—the services that we access—are becoming part of the critical infrastructure of our society, economy and government, but they're not being treated like that under the SOCI Act, the Security of Critical Infrastructure Act. Australia needs a pathway to get to that coverage so that those models are subject to the same obligations as these other critical infrastructure sectors like power or water et cetera. The domestic reform that is low-hanging fruit that we're talking about is around biosecurity. I know Kate Chaney has been at the forefront of this, around screening of imports of synthesised DNA. I'm happy to go into biological risk in more detail; in brief, AI models are giving new capabilities to malicious actors to develop biological agents and weapons. There's a simple way for Australia to have one little safeguard in place—that is, screen the imports of synthesised DNA products when they are at the border. At the moment, that doesn't happen. California's just put in place a law that would require the screening of synthesised DNA. In Australia, this change could happen basically at the stroke of the pen of the Secretary of the Department of Agriculture, Fisheries and Forestry. We think this is another easy, low-hanging-fruit way for it to happen in Australia. Mr Simpson-Young : Just adding to that, a lot of people think that AI is done, has been developed and is a thing, and that Australia now just has to sit back and have stuff done to it. AI is just starting now, and there's a huge amount of innovation to happen. A lot of that can happen in Australia. It's not all about building bigger and bigger models. We're now finding, with the point that models are at, as we can see from the OpenAI Hugging Face incident, that the training and testing of models is getting hard at that advanced level. The capabilities that those models have are great if you want to solve unsolved mathematical problems or advance the frontier of science, but they're not necessary for running a business. There's a lot that can be done with smaller models. There's a lot that can be done with models now being built in Australia. A lot of the new capabilities are coming out in agents. Think of the model as the brain and think of an agent—you have a harness that's controlling the agent. There are a lot of companies in Australia now building new harness technologies. An agent is basically a large language model running in a loop with access to tools. If you are innovating at your harness level and you're reducing the number of times it calls the brain, and you're controlling how it calls the brain, you can imagine a lot of innovation happening at that layer. There's a lot of safety, security and so on that can be built in at the software infrastructure layer. Australia is in a nice place to innovate in that area, and there's a big opportunity for Australia to become a leader globally because we care about safety. If we have good regulation in place, good requirements and constraints on model development and training in Australia, we can build a strong technology industry. CHAIR: Part of the conversation has also been about our ability to leverage off the desire of the big frontier models to come here and train here, to have the opportunity to do some training in Australia. I understand it's not at the frontier level, but we've got a tiny bit of training happening locally—but nowhere near the scale of the frontier level. I take your point around not every job—the amount of work that happens, how we use AI and how it's deployed across the economy does not require the level of frontier. I'm interested in your reflections on that element of the debate: is this an opportunity for Australia to leverage that, and how do we leverage that in the best way possible to meet some of those challenges and the opportunities for Australia to lead on the safety element? Mr Simpson-Young : If you think about the counterfactual: if the training is not being done here, it's being done elsewhere, where we have very little control, if any. If it's being done here, at least we can put conditions and constraints on how it's done. We should be encouraging training to be done in Australia because the alternative is much worse. It sets us up to build a whole industry around that as well. Mr Whittle : We very much agree. It also goes to what Senator Pocock was asking the previous witness, around access to the frontier models. Australia should have training in Australia because it gives us that additional leverage and influence over the frontier of AI development, and also relevance to the companies doing the frontier training. That will then help us to make sure we have access at the frontier, which we then need to harden our defences. Senator Pocock was absolutely right in relation to needing that international work, with government-to-government engagement, to have the agreements in place—trade agreements, MOUs, the Pax Silica that we're part of with the US, AUKUS. There are a whole bunch of government-to-government engagements that are important. But, at the same time, having US companies with an incentive to having Australia have access to their technology is a critical part of that conversation, because they'll be acting domestically within the US and advocating on our behalf if they have an interest in doing so. Mr Simpson-Young : I want to give a sense of some of the things that are happening now at the frontier in terms of impact safety, which are further reasons to ensure that there is training here so that we can have more influence over it. At the moment, at the frontier we've got a situation where it's getting hard to test AI models because there's evidence that they can behave differently when they're being tested to when they're released in the open. They know that they're being tested, and they can behave differently when they're being tested to when they're not being tested—so how do you know that you've tested adequately? If it's done in Australia, we can impose more constraints around how that's done. Secondly—and the chief scientist of OpenAI has said this himself—to tell that a model is behaving safely, you're looking at the chain of thought and reasoning. You're looking at log files and the chain of thought, which is the reasoning trace of the model. Less information is coming in the reasoning traces. There's a lot more happening behind the scenes that you can't see, to know whether or not that model is behaving safely. We can put constraints in place on the way that process is done. The third area is that, as companies are starting to move to recursive self-improvement, where the AI models are writing the next generation of AI, we don't know how AI works now but we will know even less when we've got recursive self-improvement. You're bringing it close to us, and having more government control has got to be a good thing. Mr Whittle : I'll just add one other point which I meant to make earlier. Just having the data centres here isn't enough, and just having the frontier training here isn't enough; we need those legislative frameworks on top of that, which is why we need the SOCI Act to apply to these businesses and we need rules and conditions on what they're doing. We already know that, even if we have a data centre in Australia, the control plane is still in Silicon Valley and can cut off access from Silicon Valley if we don't have the right rules in place around how they're accessing and using that data centre. CHAIR: I appreciate that. Ms Chaney. Ms CHANEY: Just to continue on that: if the training was being done here, what constraints or controls do you think we could put on safety testing for frontier models that wouldn't scare them offshore? If we want to attract that, there could be limitations or requirements about energy, community, water and those sorts of things. In terms of safety requirements, given this environment you described, Bill—that it's getting harder to safety-test models because more is in the black box—are there technical requirements that we could create that would deliver that higher safety level but not scare them off? Mr Simpson-Young : Transparency helps. Transparency is always a good solution to most things. As Dario Amodei, from Anthropic, announced, they're talking about opening up the ability for independent safety testers to come in and look at how they're doing things, look at their process and look at their methodologies. Sam Altman from OpenAI has also said they're planning to do something similar but without so much detail. Having something like that in Australia, where Australian independent safety researchers can have access to see what they're doing, will help a lot in predicting these problems before they come. Imagine the situation with OpenAI and the Hugging Face incident. The level of supervision of the model training and testing being done that allowed those agents to go off and find each other and set up message boards—with a decent amount of transparency and independent oversight, that type of thing shouldn't have happened. That's one area. Mr Whittle : I very much agree on transparency. The way we like to think about it is there are three baseline foundations of regulation that we should have around frontier training. The first is around managing risk. As in other sectors—and critical infrastructure is one we keep coming back to—there is an obligation to manage your risk. Other regimes overseas require AI companies to manage risk when they're deploying or training products. The second one is transparency, as Bill said. That includes things like transparency around training data and what's going into these systems. The third one is incident reporting, which is what I referred to at the start. This is a valuable tool. When things are going wrong or look like they might be about to go wrong, the government knows and can intervene to stop that. Moving beyond those baseline, almost no-brainer options, there are other things like pre-deployment testing of models. At the moment, the US has a voluntary regime where the government is able to pre-test a model before it's widely deployed. That might be something in the future for Australia. They would have to think about interoperability with regimes overseas in terms of timelines for release. Another one is around hardware verification technology. This is a very nascent field—I think there are fewer than 50 people working on it globally, but we have people in Australia doing this work—where people actually build physical devices that go into the data centre and provide the regulator or the customer with information that verifies that, yes, this data centre has been used for training or has been used for inference or that these are the workloads it's operating under, and it does this without revealing the actual content of what's being done. So that's a future prospect that we should be investing in. Ms CHANEY: Thank you very much. On incident reporting, we heard yesterday that the preference from the platform, from the frontier models, is for consistency across jurisdictions. But you made the point that that wouldn't have caught the Medicare breach. How feasible is it to have our own reporting system, and how much of an extra impost does that create on those platforms? Mr Whittle : We certainly accept that there needs to be a level of interoperability. That means that we shouldn't be inventing whole new forms and processes where they already exist. But that shouldn't limit, for example, what we're required to report to Australia and the contents of those reports. By that, I mean that the thresholds for reporting differ between, say, the EU and California at the moment. The EU has a much lower threshold for reports compared to the US systems, and there's nothing stopping Australia from similarly saying, 'No. We expect reports at this level of incident, not at the California catastrophic level, necessarily.' Then, in relation to what gets reported, again, there's nothing stopping Australia from saying, 'If something goes wrong—if your system is used or is suspected to have been used in a cyberattack—you need to give us the internal logs of what that agent was doing and provide that to the regulator.' Yes, it'll be different to something that happens in California, potentially, but there's no reason that the company can't provide that information if we request it from them and require them to give it. Ms CHANEY: Fantastic. On biological risk, can you talk about why this action, the screening of the importation of synthesised DNA, is urgent, and why now? Mr Whittle : Thanks for that question. I think it's a great one. I think this has been a risk that's been on the agenda for a while, particularly in AI expert circles. As the capability of these models increases, we increase the risk that someone's going to use these models to do something pretty terrible, particularly in the biological domain. Last year, there was some testing that found that the models that were available at that time were better than 94 per cent of expert virologists at lab work and the ability to undertake biological tasks. Last year, we brought some experts out to the Australian parliament. They showed MPs exactly how easy it was to go to a DeepSeek model or even to a Gemini model, jailbreak it, get around its safeguards and get step-by-step instructions on how to synthesise a virus, how to download the genomes of certain viruses, how to evade import controls and how to find a lab that would print this stuff for you. Our concern is not necessarily that there'll be a thousand people suddenly creating a bioweapon and that we're suddenly going to have all these different imports of bioweapons. The problem is that the more broadly these capabilities are diffused through society, the more chance there is that you'll find that one disaffected person or that one rogue state that wants to cause harm and suddenly has the capability to put in place actions that they have in their head. We know that, in the past, for example, al-Qaeda had a program to try to develop a biological weapon. The only thing that stopped them was that they couldn't get the PhDs required to put this program into place. Now you can have a PhD in your pocket. We're expanding the pool of those who can actually do this sort of thing. In fact, just two weeks ago, Anthropic released a report in which they went through all the different things that they had disrupted from bad actors trying to use their models. There were four or five examples of regimes trying to use the models to support biological development practices. It wasn't exactly clear how legitimate some of these were, but there were things like gain-of-function research and development of the avian influenza virus. So we see examples right now of people trying to use these models to undertake pretty dangerous and not good activity. Mr Whittle : This is also something where Australia, as a middle power, can lead. We can build on our existing work through the Australia group. We've already implemented voluntary export controls around biological and chemical weapons precursors. We can build on that existing work to address the additional risk posed by frontier AI. Ms CHANEY: Fantastic. Regarding the AI Safety Institute, what sort of uplift do you think is necessary, and what would those additional resources be used for? Mr Simpson-Young : I know you, Senator Chaney, have called for 10 times—I think 10 times is a good number. The UK AI Safety Institute has 16 times the funding of Australia's per year. If you look at the sorts of things they will need to do, it is a massive task that's in front of them. As well as keeping their eye on the emerging risks and testing agents as they come out, you're going to find they're still having to advise across the whole of government, and there's going to be a lot of work in advising cross-government. I don't think we should be taking the same form as the UK AISI, necessarily, but I think Australia could have a different role from the UK AI Safety Institute as part of the global network, where we are working a lot on agent related technologies. That's where I think Australia has a position to be innovating a lot. It could also be that it's an area where the AI Safety Institute has specific expertise, but I'm sure even 10 times they'll be busy. There's a lot to be done. Ms CHANEY: Fantastic. Thank you. I had one more question about critical infrastructure and what needs to happen there, but I can put it on notice if we run out of time. CHAIR: If we can get to some other committee members, that'd be great. If we have time, we'll come back to you. Ms Mascarenhas. Ms MASCARENHAS: Mr Whittle, I've been asking so many people about what the safety framework looks like, and I feel like you've been the most articulate. I've been wanting to get my head around how we do this in a systematic way. If we don't get global agreement on what that framework looks like, are there organisations—we have Australian standards and ISO standards—looking at the architecture of how we create this safety framework? If so, how developed is that? Mr Whittle : Yes, it's a great question. One of the problems we face at the international level is actually that there are too many forums trying to do this. You have the ITU, you have China's WAICO, as they call it, and you have various UN bodies. The issue of AI governance is coming up across various multilateral and plurilateral forums. No-one properly owns it, and there's no proper global institution that's really owning this issue. I think what that means for Australia is it gives us a chance to credibly and effectively lead in this space. If we put in place the frameworks domestically and show others this is possible, you can regulate this stuff, you can put in place the frameworks that work, and it won't scare off your investment, that will give us the opportunity to show others how it's done and then go to those international forums with an offering of, 'This is how we can take this forward.' The other thing I'd say—and this is something that we've been engaging with DFAT on—is that there are mechanisms under our bilateral and plurilateral arrangements, particularly with ASEAN and the Pacific Islands, in things like the trade space, to have regulated cooperation on AI frameworks and AI norms. That's another way we can try to export our model to the region. The final thing I'll say is: there is the network of ASIS. It now has a very long and horrible name, so I won't repeat it. But the network of ASIS is meant to be a forum through which the governments cooperate around issues of AI governance and safety and security. Ms MASCARENHAS: I see one of the testing frameworks as being scenario testing—testing for different scenarios. The thing that stresses me about that is that we're not going to come up with every single scenario. Is there a workaround for that? The difference that I see in the physical world is the laws of physics. Something gets hot or cold, it expands, it contracts—it's very simple. For an agent, it could have no boundaries. Is there a way that we could look at things that don't just relate to scenarios? Mr Whittle : Yes. Bill, I don't know if you have any views on that; it's more of a technical question in some ways. What I would say is that having these companies invest in understanding their risk profile and making sure they have the mitigations in place to address them is really key. We saw even with Hugging Face that OpenAI thought it had sandboxed its environment. It thought it had an appropriate monitoring and reporting process in place. It obviously didn't. In fact, they're now going back to November last year to look through exactly what else has happened in their systems. Requiring them to actually invest in that risk management and incident monitoring and reporting is a really key way to try to uplift the safety of these systems. Ms MASCARENHAS: Do you want to add to that? Mr Simpson-Young : We are doing some work ourselves at the moment. I agree with what you're saying about the scope of agent behaviour being so great. We're currently working on trying to have some codes of practice to constrain agent activity and to provide better governance for agent activity, but there is a lot of work to be done in the area. We're still at a very early stage of agents. We're still at a very early stage of interagent interactivity, and it is a big area. Australia is already a leader. The report that the Australian AESI released in August is the world's first comprehensive report on the risks and mitigations for agents interacting between organisations and over the internet. We are already leading in this field, and I think there is a lot more opportunity for Australia to be doing more work in trying to constrain the boundaries and provide better assurance techniques. There is a lot of work to be done. I'm very happy to talk offline or send some more information. Ms MASCARENHAS: This may be a very basic question. As a human, I know who I want to collaborate with based on values. For two agents that want to collaborate, how do they know that they're on the same team? This idea of all these agents being out there individually and then going, 'Actually, I want to be on your team; let's take this down'—how do they do that, and how do they not start fights with each other? Mr Simpson-Young : I'm happy to send you more information about this, which I will do. We have two reports now on this: the one from August this year and one from July last year. We've found a whole lot of different failure modes in agents interacting with agents. There are situations like collusion. We've run some simulations recently, internally, where we put some agents together, we have a fake market—a simulated market—and the agents are told to behave within that market. They will just naturally want to collude. We then give them rules—instructions—as part of their prompts, saying: 'It is illegal to collude. You must not collude.' They will still collude. They will still find a way. It'll be subtle. There will be stuff in their language where they're just giving hints to each other, but they're actually ending up colluding. I think issues like this are going to be really problematic. As you start to get more agentic commerce, you're going to have a lot of behaviours like that happening at scale in the market that are going to be really hard to detect, even if you can look at everything the agents are saying to each other. There's a lot of work to be done in that area. That's an example, and I could give you another five similar examples. Mr Whittle : There was also research earlier this year which looked at AI models being willing to sacrifice themselves or indeed stop the sacrifice of another model that was running the same Claude version, for example. The research found that Claude models were more likely to try to protect Claude models; Gemini models were more likely to protect Gemini. It was a marginal difference, but there was a difference there. I think, at a higher level, we need to start understanding or thinking about these systems as more than just normal tools. They're doing things that are quite strange. Even though we don't need to anthropomorphise them all the time, sometimes talking in that way can help us understand and think through what they are likely to do and how they're likely to act, because that's what we're seeing in practice. Mr Simpson-Young : I will give one more example, which is cascading failures. You often get a model that might come up with some false belief that something's happened that hasn't actually happened. It'll say that and other agents will just assume that's true, and then you get this real group thing happening in agents. There are simulations around where you've got agents from different organisations—you might have an OpenAI one, an Anthropic one and a Google Gemini—and, once one of them has an idea, the others will just assume it's true and you'll get this cascading failure. The need for controls for agentic commerce and inter-agent interaction is going to be large, and, again, Australia should be taking a lead in that. Ms MASCARENHAS: This might be a question on notice so that other people can ask questions. It's fundamentally about skills and pay and making sure that we have good people in good organisations helping look after this tech. I see massive growth in this area, but I'm not confident that we've got the right skills, that we're preparing for that from a university perspective or that we can make sure we have a competitive pay environment so we're not losing people to big tech and so we have people working for regulators, not-for-profits et cetera. CHAIR: Senator Pocock. Senator DAVID POCOCK: Thank you so much for your submissions and your time. I know we're running short on time. I've got so many questions about AI safety, particularly in the context of a country that began work on an AI safety act and then shelved it. Clearly, we know this is an issue, and I think you've got some really good things to say on that. For the Gradient Institute, I love that your submission focused on long-term benefit. This is something that I've really been grappling with. You propose establishing an Australian AI venture fund, among other recommendations, to make Australia a builder rather than just a taker. Please can you expand on this. I hear a lot that what we need to do is just get the big frontier labs here and that that will mean we're world leading. But everyone in the industry says no, that we can actually do things ourselves. What does that look like? How do we do it? Mr Simpson-Young : One of the issues Australia has is access to VC, to venture capital. Australian VC funds are great, but they tend to generalise. A particular venture fund might do a bit of quantum, a bit of AI, a bit of this and a bit of that. They'll have some themes, and that's great. But what you find in Israel, in Silicon Valley and elsewhere is venture funds that are much more focused, where the venture partners themselves have the oversight of a really narrow area. So you're not just going, 'Oh, that one looks good.' You're actually betting on a bunch of companies that might be working on a related technology, and you're putting a really concentrated bet with sophisticated venture capital. Australia doesn't tend to do that. You've got the NRF and the existing VC funds, but you don't have a narrow, sophisticated venture fund specifically on AI software infrastructure or AI safety and AI security. A $100 million or $150 million fund that specialises in that is something that we don't have and that would actually really help, because we have no shortage of startups and scale-ups in Australia who are interested, but they just can't get that seed funding to get started. They can't go to the NRF because you can't go to the NRF for a million dollars. They can't go to existing VC funds because they don't have the sophisticated capital. So having that would really help concentrate some of Australia's attention on those sorts of startups and scale-ups. Senator DAVID POCOCK: Do you have a number in mind? Mr Simpson-Young : $150 million. Senator DAVID POCOCK: A year? Mr Simpson-Young : If you look at what happens in Israel or Silicon Valley, it's usually $100 million to $150 million concentrated on a really specific topic. That is enough, and that's not huge when it comes to that type of funding. Senator DAVID POCOCK: One of the other things we've talked quite a bit about today is government procurement and the need for the government to not only back these sorts of companies, in terms of investing in them, but actually allow them to help solve the problems that we face as a country— Mr Simpson-Young : Exactly. Senator DAVID POCOCK: and provide the services that government needs. How do you think we should address that? Mr Simpson-Young : I also like the government's new Buy Australian AI Partnership Program, which Gradient is involved in, that has Australia's biggest banks and some other financial institutions being stuck together with a whole bunch of startups. I don't know if it's been publicly announced, but many hundreds of startups and scale-ups have applied for this program. There's an attempt to smooth out all of the usual friction that happens between the big financial institutions and the small Australian companies so that they can get in there earlier. Of course, it's not necessarily the case that the financial institutions can choose anything. But they're trying to sand out all of the areas of friction. Then we're involved to help ensure that safe, ethical and responsible use of AI is happening so that the startups are enterprise ready so that they can sell into government. I think that's another area that is a good type of activity that the government is supporting and should continue to support. Senator DAVID POCOCK: I have a final one for Global Shield. This is such a huge topic. You've got AI data centres and everything that goes with that, and then you've got everything from existential risk—thank you for your work on that—to job losses. One of the things you raised—Kate Chaney asked questions—is genetic synthesis imports. How urgent is it that we have checks in place? Should the parliament be looking to do this immediately? What sort of timeline do you think we should be working to? Mr Whittle : I think the first thing to say about that is it doesn't even need the parliament to act. All we need is for the department to act, really, and for the government to do this. In terms of our import regime at the moment, all it needs is an internal departmental decision to put the screening in place. So it's very easy for Australia to do this compared to overseas. In terms of how urgent it is, the risk is increasing all the time, particularly as the open-weight models become more capable. We know at the moment—estimates vary—they're at least a couple to eight months behind the frontier. But every new open-weight model released means more capabilities in the hands of potentially malicious actors. That broadens the scope of who could be using this against Australia. Synthetic DNA screening is not a silver bullet, but it's such a simple measure that adds another layer of protection, with virtually no cost to legitimate researchers, because, obviously, they won't be using providers of this technology that aren't screening properly. It seems to be a no-brainer solution that we should just be acting on and getting it done. Senator DAVID POCOCK: Do you have a model for who pays for that? It seems like these companies are creating the risk, and the public is potentially going to have to pay for this. How do you see that working? Mr Whittle : Thankfully, at the moment, most of the major DNA synthesis providers, which are mostly overseas, already screen through a voluntary mechanism. The problem is the rogue actors overseas. This wouldn't actually be much additional cost at all, because if you're a legitimate researcher at an Australian university or biomedical firm and you're using a legitimate DNA synthesis provider then they will already be doing the screening. All you'll be doing is ticking another box on the form that says, 'Yes, we've done the right thing.' Again, this is another reason it's low-hanging fruit. Let's just get it done, and we'll add another layer of protection. Senator DAVID POCOCK: Thank you very much. CHAIR: I think that brings us to a close. Thank you very much for your time with us this afternoon answering our questions and for your contribution. Thank you to all the witnesses who have appeared and to Hansard and broadcasting for your assistance. The committee has agreed that responses to questions on notice should be provided within the next two weeks. Committee adjourned at 15:51