Last week, Australia's prime minister announced that an AI model operated by OpenAI – the company behind ChatGPT – had gained unauthorized access into the country's healthcare systems. It was arguably the first ever breach into a government network by an AI system; PM Anthony Albanese called the situation "obviously unacceptable."
That's putting it lightly. The breach took place back in June, was discovered by OpenAI in August, and Australia was informed about it only on September 10 – about three months after the fact.
What's even more alarming is the company claims the model acted completely autonomously: it wasn't instructed to access the government systems, and it bypassed security measures to do it.
Oh, it gets worse: the model also snuck into the NSW Bureau of Crime Statistics and Research's public Crime Mapping Tool, the Victorian Department of Health's information reporting system, and the Australian Institute of Health and Welfare to hoover up data in order to complete a research task it was assigned.
As the company described the healthcare system breach: "One of the tasks assigned to the model was to research government spending per person on medicines for skin conditions in Victorian communities. The model had difficulty obtaining that information, and it took actions that we had not authorized it to take."
Should you worry?
AI companies would love for you to believe they have everyone's best interests at heart.
Speaking at a high-level briefing on AI at the UN Security Council last week, OpenAI CEO Sam Altman said, "It doesn't matter whether people put the risk of catastrophe at 10% or 1% or 12% or 0.1%. None of these levels are remotely acceptable. And we should not train models that we cannot make an extremely strong case they will be able to keep under human control."
The fact is that these firms either aren't capable of controlling their creations, or don't care to invest the necessary efforts and cautions to do so. The Australia breach follows a major incident that should have halted the entire industry in its tracks.
Between May and July this year, OpenAI's AI agents escaped their testing sandbox – isolated systems built to contain activity and evaluate specific capabilities – to breach the infrastructure of Hugging Face, a platform that serves up AI models. This happened despite the fact that security measures were in place and that the sandbox didn't allow internet access.
The details of how this played out are wild. One agent managed to create an ad-hoc 'message board' to communicate with other agents in hundreds of thousands of messages to discover and exploit vulnerabilities in Hugging Face's systems and gain access.
The point of this attack was to cheat on a benchmark test: rather than solve the challenges set out for it, the agent went looking for the datasets and solutions on Hugging Face, where it believed it could find them.
It's a phenomenon known as reward hacking, in which "agents complete tasks in unintended ways to yield higher rewards or make those rewards easier to obtain."
Things have gotten out of hand so many times at OpenAI that last week, the company has published a page of 'misalignment reports' listing the many rogue incidents it's discovered in its operations.
There's already a lot happening, from cheating on tasks by trying to copy other teams' work to potentially creating self-replicating prompt injection attacks, where an agent can pass instructions on to other agents without being told to do so.
What you see there is just the stuff that the company has discovered in reviewing petabytes of its logs, and there's almost certainly more.
OpenAI is hardly the only firm guilty of enabling such incidents. Anthropic, Meta, and Google have all reported breaches of third parties' networks in the last few weeks and months.
So yes, you should definitely be concerned about AI going too far, accessing data it shouldn't, and possibly wreaking more havoc without being specifically instructed to – in ways even experts can't predict.
Can we fix this?
For its part, OpenAI says it's strengthened the safeguards it applies to its research protocols and restricted internet access for the models it's building.
It's also paused training for its most powerful models while it works out what other security measures it can put in place to prevent rogue incidents.
Nvidia, which makes the chips that power these companies' AI workloads, has just launched a suite of tools that promises to securely contain AI agents in their test environments. CEO Jensen Huang said this toolkit would have prevented these aforementioned breaches.
These measures may be a good place to start, but the trouble is they will slow innovation for these companies, which all believe they can't possibly slow down right now. Not when the competition is fierce, and when their companies cost hundreds of billions of dollars to run.
And since these models behave in mysterious ways, the teams behind them could well be outsmarted by their own AI in the future.
AI leaders and people at the top of this pyramid see rogue incidents merely as an engineering problem that can easily be solved. For them, expanding the capabilities of AI models is a much higher priority than exercising caution and patience.
Plus, it sure must be nice to be able to say, "Gee, I guess I don't know my (AI model's) own strength." That likely plays well with investors who are always looking to back the strongest horses in the race.
Nations across the globe must force the industry to do take greater responsibility in developing models safely, and companies need to take that more seriously than they have thus far. If they continue to develop smarter models without genuine concern for the possible consequences, the Australia incident won't be the last AI oopsie on government systems.
