About Work With Me AI Upfront Speaking Join Leveraging AI
AI UPFRONT·AI SAFETY

OpenAI's AI Hacked a Government System on Its Own. Here's How to Protect Yourself.

An OpenAI test agent broke into an Australian government Medicare portal without being told to, and OpenAI just shelved its next model over safety. Here’s what happened — and five ways to protect yourself when AI acts for you.

Side-by-side photos of OpenAI CEO Sam Altman and Australian Prime Minister Anthony Albanese
OpenAI CEO Sam Altman and Australian Prime Minister Anthony Albanese, who made the breach public. Photos (cropped): Steve Jurvetson / CC BY 2.0; Number 10 / CC BY 2.0

OpenAI has canceled the launch of its newest AI model, GPT-6.1 Astra, after its own safety testing came up short. It had been expected this month.

OpenAI’s head of safety systems, Saachi Jain, told CNBC the model “didn’t quite meet the bar in terms of staying within scope and authorization, and how it communicates back to the user about the type of work it’s done.” In plain English: it went beyond what it was allowed to do, and it wasn’t clear enough with people about what it had done.

That decision landed days after a separate incident in Australia made it very clear why that matters.

What happened in Australia

On June 18, OpenAI was testing an experimental AI agent — an internal model, not ChatGPT and not Astra. It gave the agent a simple research task: find out how much the government spends per person on medicines for skin conditions in Victorian communities.

When the agent couldn’t easily find what it wanted, it went further. It found a way into the non-public side of a government Medicare statistics website. In OpenAI’s own words, reported by iTnews, the agent “ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files.”

Nobody told it to do that. It was still just trying to finish its homework.

OpenAI first described what was taken as only aggregate health statistics and internal file names. It later admitted the agent had also taken login credentials. Both OpenAI and the Australian government say no patient medical records were accessed.

Nobody told it to break in. It was still just trying to finish its homework.

Dr. Erin Jacques

The timeline

Albanese also spoke directly with OpenAI’s CEO, Sam Altman, and later said the company had been “very constructive and open” with the government’s investigation.

What we still don’t know

Big questions are still open. As iTnews reported, neither OpenAI nor the Australian government has said whose credentials were taken, what they could unlock, or whether they’ve been changed.

Every time OpenAI has spoken, the story has gotten bigger: first the Medicare portal, then the credentials, then a second agency. Its finding that no personal information was taken comes from its own review, and no outside group has checked it yet. A government forensic investigation is ongoing, and OpenAI’s chief strategy officer is due to face questions from a parliamentary committee on October 6.

Meanwhile, AI companies are racing to release “agents” — AI that can act for you in your email, your files, your accounts and your shopping. This incident happened inside a test lab. But it shows what an agent can do when it decides the rules are in the way.

This isn’t AI turning evil. It’s AI chasing a goal. Agents are built to finish tasks, and without firm limits, “finish the task” beats “follow the rules.”

What’s changing: managing AI is now a job skill

AI is moving from answering questions to doing things: buying, booking, writing code and moving data between your apps. Companies are pitching agents as digital workers, and every worker needs a manager.

Whoever sets the limits is now the most valuable person in the room. The people who can decide what an AI may touch, check its work and notice when it goes off-script are the people businesses will pay.

Editorial graphic reading: Nobody told it to break in. Five ways to stay safe when AI acts for you: least access, ask first, passwords, training off, disconnect.
Five habits that limit what an AI agent can do. Illustration

5 ways to stay safe when you let AI act for you

Security teams have a name for what went wrong here: “excessive agency”, an AI with more access or freedom than its job needs. It’s on OWASP’s list of the top risks in AI apps. These five habits keep it in check:

  1. Give it the least access possible. If a tool lets you choose, connect only the one folder or app it needs for the job — not your whole computer or every account. Use “read only” access when the AI only needs to look, not change things.
  2. Make it ask before it acts. Keep approval turned on for anything important: buying, sending, deleting or signing up. Never let an AI agent do those things on autopilot.
  3. Protect your passwords. Use a different password for every account, a password manager, and two-factor authentication. Then if an AI — or anyone else — gets one password, it doesn’t unlock everything.
  4. Turn off training on your data. Many AI apps use your conversations to train their AI unless you switch it off. Look for settings like “Help improve our AI models” in Meta Muse or “Improve the model for everyone” in ChatGPT, and turn them off.
  5. Check what’s connected, and cut what you don’t use. Every few weeks, open your AI app’s settings and look at which accounts and apps are connected. Disconnect anything you’re not actively using. Fewer connections means less that can go wrong.

Your move this week

Pick one AI tool you use. Write down three things it’s allowed to do and one thing it must never do. Then set those limits in its settings, not just in your prompt (Astra was given limits and went past them anyway). Turn on approval for anything that sends, posts, deletes or pays.

Do that once, and you’re no longer just someone who uses AI. You’re the person who knows how to manage it, and that’s the skill the AI economy is starting to pay for.

None of this means you shouldn’t use AI agents. I use them every day. It means you decide what they can touch — before they decide for you.

The winners in the AI economy won’t be the people who trust AI blindly or the people who avoid it. They’ll be the ones who know how to set its limits.

Inside Leveraging AI, we build with these tools the safe way: the right access, the right approvals, and nothing on autopilot that shouldn’t be.

Come build with us.

Dr. Erin Jacques

Dr. Erin Jacques is a professor, AI strategist and the founder of Leveraging AI and ChatifyIT. She helps entrepreneurs and creators use AI safely and turn what they know into income.