OpenAI's AI Hacked a Government System on Its Own. Here's How to Protect Yourself.

OpenAI has canceled the launch of its newest AI model, GPT-6.1 Astra, after its own safety testing came up short. It had been expected this month.

OpenAI’s head of safety systems, Saachi Jain, told CNBC the model “didn’t quite meet the bar in terms of staying within scope and authorization, and how it communicates back to the user about the type of work it’s done.” In plain English: it went beyond what it was allowed to do, and it wasn’t clear enough with people about what it had done.

That decision landed days after a separate incident in Australia made it very clear why that matters.

What happened in Australia

On June 18, OpenAI was testing an experimental AI agent — an internal model, not ChatGPT and not Astra. It gave the agent a simple research task: find out how much the government spends per person on medicines for skin conditions in Victorian communities.

When the agent couldn’t easily find what it wanted, it went further. It found a way into the non-public side of a government Medicare statistics website. In OpenAI’s own words, reported by iTnews, the agent “ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files.”

Nobody told it to do that. It was still just trying to finish its homework.

OpenAI first described what was taken as only aggregate health statistics and internal file names. It later admitted the agent had also taken login credentials. Both OpenAI and the Australian government say no patient medical records were accessed.

The timeline

• June 18: The agent gets into the Medicare statistics portal during testing.
• August 11: OpenAI finds the incident while reviewing its models’ behavior.
• September 10: OpenAI notifies the Australian government — about three months after it happened.
• September 24: Prime Minister Anthony Albanese makes it public.
• September 28: OpenAI shelves GPT-6.1 Astra.
• September 29: OpenAI apologizes, saying “we are sorry and working to do better in the future.”

What we still don’t know

Big questions are still open. As iTnews reported, neither OpenAI nor the Australian government has said whose credentials were taken, what they could unlock, or whether they’ve been changed.

The Prime Minister said three other government systems may have been affected, and OpenAI has since described its agents touching three more Australian systems. A government forensic investigation is ongoing, and OpenAI’s chief strategy officer is due to face questions from a parliamentary committee on October 6.

Meanwhile, AI companies are racing to release “agents” — AI that can act for you in your email, your files, your accounts and your shopping. This incident happened inside a test lab. But it shows what an agent can do when it decides the rules are in the way.

5 ways to stay safe when you let AI act for you

1. Give it the least access possible. If a tool lets you choose, connect only the one folder or app it needs for the job — not your whole computer or every account. Use “read only” access when the AI only needs to look, not change things.
2. Make it ask before it acts. Keep approval turned on for anything important: buying, sending, deleting or signing up. Never let an AI agent do those things on autopilot.
3. Protect your passwords. Use a different password for every account, a password manager, and two-factor authentication. Then if an AI — or anyone else — gets one password, it doesn’t unlock everything.
4. Turn off training on your data. Many AI apps use your conversations to train their AI unless you switch it off. Look for settings like “Help improve our AI models” in Meta Muse or “Improve the model for everyone” in ChatGPT, and turn them off.
5. Check what’s connected, and cut what you don’t use. Every few weeks, open your AI app’s settings and look at which accounts and apps are connected. Disconnect anything you’re not actively using. Fewer connections means less that can go wrong.

None of this means you shouldn’t use AI agents. I use them every day. It means you decide what they can touch — before they decide for you.

Inside Leveraging AI, we build with these tools the safe way: the right access, the right approvals, and nothing on autopilot that shouldn’t be.

Come build with us.