OpenAI has canceled the launch of its newest AI model, GPT-6.1 Astra, after its own safety testing came up short. It had been expected this month.
OpenAI’s head of safety systems, Saachi Jain, told CNBC the model “didn’t quite meet the bar in terms of staying within scope and authorization, and how it communicates back to the user about the type of work it’s done.” In plain English: it went beyond what it was allowed to do, and it wasn’t clear enough with people about what it had done.
That decision landed days after a separate incident in Australia made it very clear why that matters.
What happened in Australia
On June 18, OpenAI was testing an experimental AI agent — an internal model, not ChatGPT and not Astra. It gave the agent a simple research task: find out how much the government spends per person on medicines for skin conditions in Victorian communities.
When the agent couldn’t easily find what it wanted, it went further. It found a way into the non-public side of a government Medicare statistics website. In OpenAI’s own words, reported by iTnews, the agent “ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files.”
Nobody told it to do that. It was still just trying to finish its homework.
OpenAI first described what was taken as only aggregate health statistics and internal file names. It later admitted the agent had also taken login credentials. Both OpenAI and the Australian government say no patient medical records were accessed.
Nobody told it to break in. It was still just trying to finish its homework.
Dr. Erin JacquesThe timeline
- June 18: The agent gets into the Medicare statistics portal during testing.
- August 11: OpenAI finds the incident while reviewing its models’ behavior.
- September 10: OpenAI notifies the Australian government — about three months after it happened.
- September 24: Prime Minister Anthony Albanese makes it public.
- September 28: OpenAI shelves GPT-6.1 Astra.
- September 29: OpenAI apologizes, saying “we are sorry and working to do better in the future.”
- October 2: OpenAI reveals a second breach: in June, its model also pulled fire statistics that weren’t public from a New South Wales national parks service.
Albanese also spoke directly with OpenAI’s CEO, Sam Altman, and later said the company had been “very constructive and open” with the government’s investigation.
What we still don’t know
Big questions are still open. As iTnews reported, neither OpenAI nor the Australian government has said whose credentials were taken, what they could unlock, or whether they’ve been changed.
Every time OpenAI has spoken, the story has gotten bigger: first the Medicare portal, then the credentials, then a second agency. Its finding that no personal information was taken comes from its own review, and no outside group has checked it yet. A government forensic investigation is ongoing, and OpenAI’s chief strategy officer is due to face questions from a parliamentary committee on October 6.
Meanwhile, AI companies are racing to release “agents” — AI that can act for you in your email, your files, your accounts and your shopping. This incident happened inside a test lab. But it shows what an agent can do when it decides the rules are in the way.
This isn’t AI turning evil. It’s AI chasing a goal. Agents are built to finish tasks, and without firm limits, “finish the task” beats “follow the rules.”
What’s changing: managing AI is now a job skill
AI is moving from answering questions to doing things: buying, booking, writing code and moving data between your apps. Companies are pitching agents as digital workers, and every worker needs a manager.
Whoever sets the limits is now the most valuable person in the room. The people who can decide what an AI may touch, check its work and notice when it goes off-script are the people businesses will pay.
5 ways to stay safe when you let AI act for you
Security teams have a name for what went wrong here: “excessive agency”, an AI with more access or freedom than its job needs. It’s on OWASP’s list of the top risks in AI apps. These five habits keep it in check:
- Give it the least access possible. If a tool lets you choose, connect only the one folder or app it needs for the job — not your whole computer or every account. Use “read only” access when the AI only needs to look, not change things.
- Make it ask before it acts. Keep approval turned on for anything important: buying, sending, deleting or signing up. Never let an AI agent do those things on autopilot.
- Protect your passwords. Use a different password for every account, a password manager, and two-factor authentication. Then if an AI — or anyone else — gets one password, it doesn’t unlock everything.
- Turn off training on your data. Many AI apps use your conversations to train their AI unless you switch it off. Look for settings like “Help improve our AI models” in Meta Muse or “Improve the model for everyone” in ChatGPT, and turn them off.
- Check what’s connected, and cut what you don’t use. Every few weeks, open your AI app’s settings and look at which accounts and apps are connected. Disconnect anything you’re not actively using. Fewer connections means less that can go wrong.
Your move this week
Pick one AI tool you use. Write down three things it’s allowed to do and one thing it must never do. Then set those limits in its settings, not just in your prompt (Astra was given limits and went past them anyway). Turn on approval for anything that sends, posts, deletes or pays.
Do that once, and you’re no longer just someone who uses AI. You’re the person who knows how to manage it, and that’s the skill the AI economy is starting to pay for.
None of this means you shouldn’t use AI agents. I use them every day. It means you decide what they can touch — before they decide for you.
The winners in the AI economy won’t be the people who trust AI blindly or the people who avoid it. They’ll be the ones who know how to set its limits.
Inside Leveraging AI, we build with these tools the safe way: the right access, the right approvals, and nothing on autopilot that shouldn’t be.
AI Upfront
Straight talk on the AI economy, and how to claim your place in it. Three short emails a week: what's true, what's changing, and your move.