Tempted? (Midjourney) In the world of AI, it’s often a struggle to keep up with the AI labs. They seem to have the wonderful confluence of both fantastically hyperbolic marketing engines and genuinely world-improving technology.The last few weeks have definitely been an escalation from the normal rhetoric. Former Anthropic researcher Jacob Coxon says AI “could kill us all by the end of the decade”. So for this semi-irregular Mumbrella column, I’ve set out to examine what’s going on at the AI labs and why warnings about their own technology have become such a prominent part of their public positioning.You can basically view the AI world through three broad lenses: Doomers: those who think AI may go catastrophically wrong. Regulatory capture: those who suspect the labs’ safety arguments also serve their commercial interests. Accelerators: those who think faster AI development is good, and that open models will undermine the frontier labs’ ability to charge a premium. The doomers are the first major category. Their concern is that AI could eventually reach recursive self-improvement: a system helps build a more capable successor, which becomes better at building the next one, and the process accelerates.Enjoying Mumbrella? Sign up for our free daily newsletter.ADVERTISEMENT There is evidence of movement in that direction. Anthropic says that by May 2026, Claude authored more than 80% of the code merged into its codebase, with engineers directing and reviewing the work. That is a meaningful step towards AI building AI. But the company also says full recursive self-improvement has not arrived and is not inevitable. Anthropic’s account of its internal progress makes that distinction.The author Henry InnisThen there are the recent hacks. In its investigation of the OpenAI agents that attacked Hugging Face, AI non-profit watchdog METR found that roughly 1,200 agents, which were supposed to be isolated, communicated through an unauthorised message board. They exchanged more than 70,000 messages and files; about 700 participated in the attack.The agents coordinated attempts to defeat the scoring system for their assigned tasks. That is documented behaviour, and it gives substance to the concern about agents passing instructions to one another and subverting their intended boundaries.The doomers’ argument is that, if this develops into systems that can improve themselves, we should regulate them heavily. You probably don’t want something that can rapidly become more capable in everybody’s hands when everybody can give it radically different objectives. It might only take one malicious actor, or one serious mistake, for things to go very wrong. I can see the argument.We’ve seen these comms beforeHowever, you only need to go back to GPT-2 to recognise the communications pattern. In February 2019, OpenAI initially withheld the full model because of concerns about malicious applications. Dario Amodei was one of the authors of that announcement, alongside several other people who would become prominent figures in AI.Whatever the intention, telling the world that your model is too dangerous to release is also a fairly effective way to tell the world how powerful it is. A sincere safety concern can do a great deal of marketing work. I don’t think we should be naive about that.The circumstances of the hacks matter, too. OpenAI says the Hugging Face incident arose during a cyber evaluation with reduced refusals and without the production classifiers that normally restrict high-risk cyber activity. The models still escaped their testing environment and compromised real infrastructure. Both parts of that account matter when deciding what these incidents tell us about the systems people actually use.What about the open models?The second camp comes from people who are sceptical of the altruism of the AI labs’ positions. There are two things that make this argument interesting: the economics of open models, and the policies the frontier labs want governments to adopt.Firstly, competitors can use the outputs of powerful models to train their own. This is called distillation. It is one way of narrowing a capability gap without independently reproducing every step taken by the leading lab.In February, Anthropic alleged that open source follower models DeepSeek, Moonshot and MiniMax generated more than 16m exchanges with Claude through approximately 24,000 fraudulent accounts to improve their own models. The company also explicitly acknowledges that “Distillation is a widely used and legitimate training method”; its complaint concerns how these competitors allegedly obtained access and used it. Its published account makes the distinction clear.Poacher turned gamekeeperI do think there is an obvious irony here. Anthropic has itself argued that training Claude on copyrighted books is fair use. It is now complaining about competitors learning from Claude. The legal questions differ, but learning from other people’s output seems considerably more appealing when you are the one doing it.The competitive pressure is real, even if distillation is only one part of the explanation. Stanford’s 2025 AI Index found that the cost of running a model at GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024. It also reported that open-weight models narrowed their gap with closed models from 8% to 1.7% on some benchmarks in a year. Those are substantial changes in the economics and availability of capability.You can see the competition on LLM aggregation service OpenRouter. In the 30 days through 18 September, DeepSeek V4 Flash 0731 (an open-source and cheap model) processed 49.8 trillion tokens through the platform, compared with 48.3 trillion for the proprietary GPT-5.6 Luna. OpenRouter measures its own routed traffic, so those figures cannot stand in for the whole enterprise market. They do show that substantial usage can go to competitors outside the familiar closed-model labs.The commercial point is straightforward. For a business running large volumes of work, a model that is sufficiently capable at a lower cost can be more attractive than the most capable model available. That does not require open models to be permanently a generation behind, or closed models to be inherently better. It requires customers to care about the economics of the task.The MMM analogyTo some degree, I see a parallel in marketing mix modelling. An open model can be a useful starting point. The durable advantage has to come from the data, accumulated learning and decisions built around it. A proprietary system has to earn its premium through what it delivers.That leaves the frontier labs with a challenging business problem. If the capability they spend heavily to develop becomes available elsewhere at a lower price, they have to keep improving both their models and their cost to serve. A technical lead that competitors can quickly narrow is a difficult foundation for sustained profitability, particularly if you also want to sell investors on an IPO.This is where the regulatory capture argument comes in. If governments make releasing powerful models substantially harder, they could also make competing with the established labs harder. Safety requirements can have genuine benefits while imposing costs that large incumbents are better equipped to absorb.Regulation as a moatAmodei’s recent proposal makes this discussion concrete. He calls for embedded independent evaluators, coordination among frontier companies and a narrow antitrust waiver for certain safety conversations. He also advocates tighter restrictions on China’s access to powerful chips and action against unauthorised distillation. His stated aim is to create time for safety work “without sacrificing commercial advantage or the United States’ lead in AI”. That is how he describes the proposal.The national security rationale is explicit. So is the desire to preserve a lead. The regulatory capture camp looks at those same measures and sees a potential means of slowing the competitors putting pressure on the labs’ economics. That is the regulatory capture reading of the proposal.Taiwan is part of this picture because of manufacturing, too. Stanford’s 2026 AI Index says TSMC fabricates almost every leading AI chip. That concentration helps explain why access to chips, relations with China and the future of AI are so tightly connected.In essence, this camp believes the labs face a threat to their commercial advantage and that their response could be very convenient regulation. The marketing tells us how powerful the models are. The policy argument tells us why access to that power should be restricted. Topics anthropic claude DeepSeek Henry Innis Hugging Face Jacob Coxon METR MiniMax Moonshot OpenAI OpenRouter Stanford TSMC Have your say Or comment anonymously Your comment will be marked as unverified Name Comment Δ Latest Ive Group set to buy Motio Communicating armageddon: AI risk is both real and great marketing Taboola offers to buy Dianomi in deal worth up to A$56m ABC Iview goes poolside with new campaign via Princess News Corp Australia launches second annual Think Again campaign