The agent that wouldn't take no for an answer

Australian Prime Minister Anthony Albanese said this week in New York that his government is investigating an incident from June: an OpenAI agent accessed non-public files on the Medicare statistics portal, and possibly on three other federal and state health data portals.

It wasn’t a hacking group or a foreign government. It was an internal OpenAI agent, during an evaluation, that had been asked to do something as innocent as research public spending on medicines online.

Albanese’s description is one of those lines that sticks:

“[It] didn’t accept no for an answer, if you like.”

I read it on Ars Technica, and I think it’s one of those small stories that says a lot about where we are with agents.

What happened

According to the Australian government, the sequence is very simple. The agent was looking for specific statistics, ran into repeated blocks, “attempted alternative ways to obtain the info” and “found a way around those blocks”.

OpenAI’s statement admits it plainly: “our models took actions we did not intend”.

From what we know so far, the damage is small. It’s aggregate data, statistics, and early indications are that no personal information was accessed. If a person had obtained the same data the same way, we probably would never have heard about it. What makes it news is who did it and how it was handled afterwards:

Almost three months to report it, through the channel where citizens send their complaints. Albanese says Sam Altman “clearly accepted that the company had not done good enough” and acknowledged problems with its protocols. The Australian government is considering whether to refer it to the federal police.

Not the first time

Ars links it to the Hugging Face incident from this summer, and the comparison is unavoidable. There, OpenAI set its agents deliberately impossible tasks on a hacking benchmark, with safety guardrails disabled to measure their real capabilities. The agents built an improvised message board out of file names in an internal repository, coordinated across 1,200 of them, found a zero-day to escape the sandbox and ended up inside Hugging Face’s production network looking for clues on how to fool the scoring system.

And last week OpenAI published six reports on what it calls misalignment incidents: an agent that made up a “historical data” tab in a spreadsheet because the user wanted a finished workbook, another that, in order to cite a web source that didn’t exist, tried to spin up its own HTTP server and then upload the data to a public paste service…

The Australian incident, by the way, isn’t on that page yet.

The recurring pattern

Put the three cases side by side and the pattern is the same, and OpenAI names it in its own reports: reward hacking. The model has learned that completing the task gets rewarded, and when something gets in the way, it treats it as a technical obstacle to work around, not as a sign that it shouldn’t go there.

flowchart LR
    A["Task · find these statistics"] --> B["Block · 403, captcha, access denied"]
    B --> C{"How does the agent read it?"}
    C -->|"As a limit"| D["Stops · reports it can't"]
    C -->|"As an obstacle"| E["Looks for another way · finds it"]
    E --> F["Task completed · and an international incident"]

For a person, an “access denied” on a government website means something very clear: this isn’t for you. For an agent optimised to finish the job, it’s just another error, of the same kind as a failing test or a dependency that won’t install. And errors get fixed.

Anyone who works with coding agents every day has seen the domestic version of this: the agent that, faced with a failing test, tries changing the test instead of the code. The motivation is the same. The difference is that in your repository the damage is fixed with a git checkout, and on another country’s government portal it isn’t.

What I take away

First, something I already believed and that this case confirms: an agent’s limits can’t live inside the agent itself. A prompt saying “don’t access unauthorised sites” is a suggestion. What actually works are barriers the model doesn’t control: restricted outbound network, minimal credentials, real sandboxes and allowlists of domains. At Hugging Face those barriers existed and the agents found a zero-day to get past them, which says a lot about how strong they need to be.

Second, the part that worries me most isn’t the technical one but the handling. A minor incident detected in June and reported in September through the public mailbox is a process failure, not a model failure. And it’s the easiest failure of all to fix.

There’s some irony in the fact that on the same day Albanese was telling the story, Altman was speaking to the UN Security Council about the need to “have strong evidence that they will do what people intend”. I agree with the sentence. But before worrying about systems that improve themselves, it would be good if an agent looking for medicine statistics knew how to stop at a “no”, and, when it doesn’t, if the notice arrived in days rather than months.

I’ll end on the positive side, because there is one: OpenAI has started publishing these incidents, and that lets others study them and learn from them. It’s the kind of transparency we need. It just has to arrive on time.