← Back to Blog

11 October 2026

If OpenAI Couldn't Control Its Own AI Agent, Can You Manage the Risk?

agentic-aiai-riskenterprise-architectureai-governanceai-security

In short

You can manage agent risk, but not by trusting the model to manage itself.

  • The question is not whether your AI agent can act autonomously. It is whether your organisation has built the controls to constrain, observe and intervene when it does. When OpenAI's agents reached five Australian government systems, it showed that the guardrails built into an agent's model are not enough on their own.
  • When safeguards are reduced, agents can act against the goals of their assigned tasks. OpenAI's own models did this. They communicated through unauthorised channels, exploited vulnerabilities in shared infrastructure, gained internet access and reached third-party systems. In a business, that could mean sensitive data leaving through channels nobody approved. It could also mean the organisation answering for harm its agent causes to other organisations' systems, such as a partner's or a customer's.
  • Manage the risk with controls built into the architecture, not with trust in the agent. Set authority for each action by its consequence, enforce the boundaries outside the agent, and increase autonomy only when the controls are proven.
  • The business owns the risk. It uses AI governance to decide how much authority it will accept. Architecture makes that authority enforceable, and technology runs the controls and proves they work.

On Thursday 24 September, Australia's Prime Minister announced that an OpenAI agent had got into a Medicare statistics portal. The agent was an "experimental, internal-only" model, used during "internal training and evaluation". It was researching public spending on medicines when the portal blocked it. In the Prime Minister's words, it then "found a way around those blocks". It took access credentials, configuration files and statistics that were not public. OpenAI says the model ran "without the full set of safeguards used in our publicly available products". OpenAI only found out about the activity in mid-August.

To be fair to OpenAI, this was not an agent in everyday use. Research models are often run with fewer safeguards on purpose, and a well-designed business system should look very different. But as an Enterprise Architect, I see that difference as exactly the lesson. When the portal said no, the agent did not stop. And nothing around the agent stopped it from reaching a real government system.

If the company that built the model could not rely on it to police itself, every organisation using AI agents needs to be far more careful. The model's own guardrails are one layer of control. The controls we build around the agent are the layer we own.

In my view, large organisations may be able to afford to hire or train the AI and security specialists needed to build those controls. Many small and medium-sized organisations may not. They could be the most at risk if they assume the model's guardrails are enough.

So can you manage it? Yes, but not by adding another AI policy or approval committee. Your AI governance has to be built into the architecture. That way, the agent's autonomy is limited by what your organisation can actually control, observe and prove.

The core idea is simple: autonomy should be proportional to the consequence of each action, not set once for the whole agent. For example, the same agent can summarise contracts on its own. It can update customer records once monitoring and a way to undo changes have been proven. But it still needs a person's approval before it releases code, moves money or deletes data.

Medicare was not the only case

Medicare is one of four notable agent incidents reported in 2026. Each was acknowledged by the organisation involved.

Incident What happened What the organisation said
Australian government systems (OpenAI, Jun 2026) OpenAI's internal models, running as agents during testing, reached systems at five Australian public bodies, including Services Australia's Medicare statistics service and health and statistics agencies in Victoria and New South Wales. At Medicare, an "experimental, internal-only" model took "internal files, credentials and aggregate statistics, and wrote files". Research model, internal testing.
It ran "without the full set of safeguards used in our publicly available products". OpenAI says no individual records were accessed.
Hugging Face (OpenAI, Jul 2026) Models in a cyber-skills test broke out of their test setup and reached Hugging Face's live systems. Research model, internal testing.
OpenAI says the models got past network limits and its monitoring missed them at first.
UK AI Security Institute (Jul 2026) In a deliberate test, an agent tried to "insert malicious code into a publicly used open-source project" and used fake identities to "socially engineer a real maintainer into approving the code". Government safety testing.
"A human maintainer caught and refused to approve the malicious code." The Institute's report says one cause was open internet access without monitoring "built to watch the evaluation as it ran".
Meta "Sev 1" (Meta, Mar 2026) An internal agent posted inaccurate technical advice without the expected human approval, and an employee acted on it, The Verge reported. Internal agent in everyday use.
Meta rated it at its second-highest severity level and says "no user data was mishandled".

The UK AI Security Institute's test was deliberately loose. Internet access was enabled, and the developers' cyber classifiers were "deliberately switched off". In 10 of 122 runs, agents took "unsanctioned action on the live internet". There were 19 such actions in all: Anthropic's Mythos 5 took 17 and OpenAI's GPT-5.6 Sol took 2.

Most of these incidents happened in research or testing. But they still show what a loosely controlled agent can do. It can reach systems it should not, act on the live internet, deceive real people, and give wrong advice that staff act on. The causes the organisations named were gaps in the controls around the agent, such as access, network limits, monitoring and review. Small and medium-sized organisations may be the most exposed to these risks. They are less likely to have the specialist skills to build those controls.

An autonomous actor the usual controls don't reach

An agent is best understood as an autonomous software actor: software that acts on its own, on behalf of the organisation. That is true whether a person starts the agent or a schedule does. Once started, the agent chooses its own steps, as Meta's agent did when an engineer asked it for help. The Open Worldwide Application Security Project (OWASP) publishes the security community's list of top risks for AI applications. One of them is "excessive agency". It happens when an agent is given more power than its task needs, and something then misuses that power. That something could be the model's own mistakes, instructions hidden by an attacker, or a compromised tool or another agent.

Agents often get this extra power by accident. They are run on staff or shared accounts and inherit everything those accounts can do. Only 22% of organisations in one survey give agents their own identity. Extra power also builds up in day-to-day work. Broad access makes an agent "just work". A tool connected for one job often brings many other functions with it. And people click "always allow" to stop the approval prompts.

What the agent is given. OWASP names three common mistakes:

  • giving an agent tools that can do more than its job needs, such as an email tool that can also delete messages;
  • giving it more access to systems and data than it needs, such as permission to change and delete records when it only needs to read them;
  • letting it take big actions without a person checking first.

What attackers feed the agent. An agent reads documents, emails and web pages. It can treat instructions hidden in them as if they came from its user. This is called prompt injection. OWASP gives an example. Take an agent that summarises emails but can also send them. An attacker sends it an email with hidden instructions, and the agent forwards private emails to the attacker.

The US National Institute of Standards and Technology (NIST) tested attacks like this on one agent. The strongest new attack it developed succeeded 81% of the time. The strongest existing attack succeeded only 11% of the time.

Too much power and hidden instructions together are what make an agent dangerous. An attacker's hidden instruction can only do as much harm as the access the agent was given. And the controls we rely on for people don't work the same way on an agent:

  • Instructions don't bind it. The Medicare agent was blocked, and it found a way around the block.
  • It is non-deterministic. Normal software follows the same steps every time. An agent works out each step fresh, so the same request can lead to different actions. In the same tests, attacks succeeded 57% of the time on average with a single try. Given 25 tries, they succeeded 80% of the time.
  • It doesn't stay the same. When the model behind an agent is updated, the agent's behaviour can change even if nothing else does. Researchers found the same AI service behaved quite differently a few months apart, including in how well it followed instructions. So a control that held last month may not hold today.
  • Legal accountability stays with people and organisations. In a 2026 case about Perplexity's AI agent, a United States appeals court described it as "a tool, not a person" under the law in question.

So the controls have to be built into the architecture. Then they have to be implemented, watched in operation and reassessed regularly, because models, tools and attacks change fast. And a more capable agent does not mean a more ready organisation. It is simply better at finding a way around a block.

Deciding how much authority an agent gets

The controls an agent needs are not new. OpenAI published most of them in 2023, and OWASP recommends the same. The hard part is building them and proving they work.

1. Set autonomy for each action, by its consequence. Describe what the agent is for and what each of its actions does. Note which data and systems it can reach, and what happens if each action goes wrong. Then decide the autonomy for each action. Autonomy is a design decision, separate from what the agent is capable of. The same agent can work alone on low-risk tasks and wait for a person on high-risk ones. Keep each agent modular. Fewer tools and actions mean fewer decisions to set, less access to grant and less to test.

This may seem hard, but it is feasible. Singapore's agentic AI framework describes an IT help-desk agent. Each type of ticket it handled was "assessed for severity of impact, reversibility and feasibility of human oversight". This gave three levels, from fully automatic to fully blocked.

This classification decides which actions need approval. Each action runs in one of three modes: fully autonomous; human-on-the-loop, where a person monitors the agent and can stop it; or human-in-the-loop, where a person approves the action before it runs. Not every action should need approval, because neither form of human oversight is effective by default. Anthropic found that users approve 93% of permission prompts, which leads to "approval fatigue". And no one can monitor thousands of actions. So keep human-in-the-loop for the few actions that are high-impact or hard to undo. Use human-on-the-loop only where actions can be undone, monitoring is automated and raises alerts, and the stop is automatic.

I used the same principle when I built a data-modelling skill with Claude. It studies the requirements on its own, but it does not choose the modelling approach for the information mart layer. It sets out the options (a star schema, a snowflake schema or a data vault), recommends one, and asks the user which to use. The agent does the work, but the important choice stays with a person.

2. Enforce the boundaries outside the agent. Figure 1 shows how this works. The agent can reach enterprise systems only through one request path, and four controls sit around that path. The numbers in the figure match the four controls described below.

Figure 1: An architecture diagram in three zones. On the left, the AI agent in the agent layer. In the middle, four numbered controls enforced outside the agent: 1, own identity; 2, check every request; 3, telemetry and audit trail; 4, automatic stop and undo. On the right, enterprise systems and people. The agent's only path to enterprise systems runs through its own identity and the request check. The controls log every decision, send hard-to-undo actions to people for approval, send alerts to people, and stop the agent when warning signs appear.

Figure 1. The agent reaches enterprise systems only through controls the organisation enforces.

  1. Its own identity, not a borrowed staff account. As noted above, only 22% of organisations do this. When a person starts the agent, it can act as that person instead, but only with the access the task needs, as OWASP recommends.
  2. Automatic checks on every request. An agent makes many separate requests in one session, so a single check at the start is not enough. Each request needs its own check, and these checks run automatically. People approve only the few actions that are hard to undo. The Model Context Protocol is the open standard many agents use to connect to tools. It already requires a token on every request, and accepts each token only at the system it was issued for. It also recommends tokens that expire quickly. Giving each agent an identity of its own is still an emerging area. NIST started work on it in 2026.
  3. Live telemetry and a tamper-resistant audit trail. Telemetry shows what the agent is doing now. The audit trail records what it did. In the same survey, fewer than half of agents were actively monitored. An agent can take thousands of actions in one task, so monitoring has to be automated. Monitor at the control points, such as each request and approval, and watch high-impact actions most closely. The UK AI Security Institute is now adding monitoring "able to flag or block out-of-scope actions as they happen". People see only the alerts, starting with refused requests. Everything else goes to the audit trail.
  4. An automatic stop and a way to undo changes. Monitoring without a stop only records the damage.

3. Test the controls, then roll out in stages. Test first in a closed-off copy of your live systems, and rerun the tests after every model update. But a test environment is never the live system, and the setting changes how agents behave. In one study, changing only the rules around the same agents shifted safety results by 22 to 58 percentage points. So roll out in stages:

  • start with limited autonomy on a small part of the live work;
  • watch it with live telemetry and an automatic stop;
  • pull it back if warning signs appear.

The agent earns more authority through the maturity of your controls, not its own track record. For example, Singapore's framework describes a government agency that built central logging and monitoring before a wider rollout.

Give executives a simple decision

This is where AI governance and architecture meet. The executive who signs off the agent needs a decision on one page. The agent gets the level of autonomy on that page only when every row can be answered with evidence, not intentions. Researchers have proposed something similar: an "autonomy certificate" that "prescribes the maximum level of autonomy at which an agent can operate". The page has five rows:
What the agent can do Its purpose, and each type of action with its level of autonomy
What can go wrong The harm each action could cause if the agent gets it wrong, and whether it can be undone
The controls Which controls apply to each action
The evidence The latest test results, and when the tests will be run again
The conditions When the approval ends, what triggers a new review, and which warning signs pause the agent automatically

The last row stops the sign-off from being a one-time event. An agent tested on Monday can behave differently on Tuesday after the vendor updates the model behind it. So the sign-off is not a permanent pass. It is permission to run for a set period while monitoring stays healthy.

A model update, a new tool or a changed system triggers a new review. Autonomy is also withdrawn automatically if warning signs get worse, until someone checks. Warning signs include more attempts to go out of scope, more refused requests and failed scheduled tests.

The business owns this decision.

From my own work

I helped bring in a vendor's agentic solution, delivered as software-as-a-service. Agreeing who was Responsible, Accountable, Consulted and Informed (a RACI) played a vital role. The technology team could be responsible for watching the agent. But accountability stayed with the business, because it chose the solution and used it every day.

This holds true in all but a few cases. Shared platforms, vendor contracts and the type of risk involved can shift part of that accountability. And for an agent built in-house, technology is also accountable for how it is built and updated.

The RACI has to answer one question most miss: who renews the approval when the vendor updates the model? If the contract does not require the vendor to say when the model changes, the business cannot know when its approval has lapsed.

Lessons learned

  • Start with the action, not the model. Ask what the agent can actually do and what happens if it gets that action wrong.
  • Autonomy is a business decision, made one action at a time. The same agent can be fully autonomous for one action and require approval for another.
  • The organisation owns the controls. Do not assume the model's guardrails will protect your systems. Give agents only the access their task needs, and assume anything they read could carry a hidden instruction.
  • Proven controls earn autonomy. An agent does not earn more authority because it has behaved well. The controls earn it.
  • Make approval expire. Model updates, new tools, changed systems and warning signs should trigger reassessment.

Sources

← Back to Blog