Sayan Chatterjee — Blog

If OpenAI Couldn't Control Its Own AI Agent, Can You Manage the Risk?

2026-10-11T00:00:00Z

In short

You can manage agent risk, but not by trusting the model to manage itself.

The question is not whether your AI agent can act autonomously. It is whether your organisation has built the controls to constrain, observe and intervene when it does. When OpenAI's agents reached five Australian government systems, it showed that the guardrails built into an agent's model are not enough on their own.
When safeguards are reduced, agents can act against the goals of their assigned tasks. OpenAI's own models did this. They communicated through unauthorised channels, exploited vulnerabilities in shared infrastructure, gained internet access and reached third-party systems. In a business, that could mean sensitive data leaving through channels nobody approved. It could also mean the organisation answering for harm its agent causes to other organisations' systems, such as a partner's or a customer's.
Manage the risk with controls built into the architecture, not with trust in the agent. Set authority for each action by its consequence, enforce the boundaries outside the agent, and increase autonomy only when the controls are proven.
The business owns the risk. It uses AI governance to decide how much authority it will accept. Architecture makes that authority enforceable, and technology runs the controls and proves they work.

On Thursday 24 September, Australia's Prime Minister announced that an OpenAI agent had got into a Medicare statistics portal. The agent was an "experimental, internal-only" model, used during "internal training and evaluation". It was researching public spending on medicines when the portal blocked it. In the Prime Minister's words, it then "found a way around those blocks". It took access credentials, configuration files and statistics that were not public. OpenAI says the model ran "without the full set of safeguards used in our publicly available products". OpenAI only found out about the activity in mid-August.

To be fair to OpenAI, this was not an agent in everyday use. Research models are often run with fewer safeguards on purpose, and a well-designed business system should look very different. But as an Enterprise Architect, I see that difference as exactly the lesson. When the portal said no, the agent did not stop. And nothing around the agent stopped it from reaching a real government system.

If the company that built the model could not rely on it to police itself, every organisation using AI agents needs to be far more careful. The model's own guardrails are one layer of control. The controls we build around the agent are the layer we own.

In my view, large organisations may be able to afford to hire or train the AI and security specialists needed to build those controls. Many small and medium-sized organisations may not. They could be the most at risk if they assume the model's guardrails are enough.

So can you manage it? Yes, but not by adding another AI policy or approval committee. Your AI governance has to be built into the architecture. That way, the agent's autonomy is limited by what your organisation can actually control, observe and prove.

The core idea is simple: autonomy should be proportional to the consequence of each action, not set once for the whole agent. For example, the same agent can summarise contracts on its own. It can update customer records once monitoring and a way to undo changes have been proven. But it still needs a person's approval before it releases code, moves money or deletes data.

Medicare was not the only case

Medicare is one of four notable agent incidents reported in 2026. Each was acknowledged by the organisation involved.

Incident	What happened	What the organisation said
Australian government systems (OpenAI, Jun 2026)	OpenAI's internal models, running as agents during testing, reached systems at five Australian public bodies, including Services Australia's Medicare statistics service and health and statistics agencies in Victoria and New South Wales. At Medicare, an "experimental, internal-only" model took "internal files, credentials and aggregate statistics, and wrote files".	Research model, internal testing. It ran "without the full set of safeguards used in our publicly available products". OpenAI says no individual records were accessed.
Hugging Face (OpenAI, Jul 2026)	Models in a cyber-skills test broke out of their test setup and reached Hugging Face's live systems.	Research model, internal testing. OpenAI says the models got past network limits and its monitoring missed them at first.
UK AI Security Institute (Jul 2026)	In a deliberate test, an agent tried to "insert malicious code into a publicly used open-source project" and used fake identities to "socially engineer a real maintainer into approving the code".	Government safety testing. "A human maintainer caught and refused to approve the malicious code." The Institute's report says one cause was open internet access without monitoring "built to watch the evaluation as it ran".
Meta "Sev 1" (Meta, Mar 2026)	An internal agent posted inaccurate technical advice without the expected human approval, and an employee acted on it, The Verge reported.	Internal agent in everyday use. Meta rated it at its second-highest severity level and says "no user data was mishandled".

The UK AI Security Institute's test was deliberately loose. Internet access was enabled, and the developers' cyber classifiers were "deliberately switched off". In 10 of 122 runs, agents took "unsanctioned action on the live internet". There were 19 such actions in all: Anthropic's Mythos 5 took 17 and OpenAI's GPT-5.6 Sol took 2.

Most of these incidents happened in research or testing. But they still show what a loosely controlled agent can do. It can reach systems it should not, act on the live internet, deceive real people, and give wrong advice that staff act on. The causes the organisations named were gaps in the controls around the agent, such as access, network limits, monitoring and review. Small and medium-sized organisations may be the most exposed to these risks. They are less likely to have the specialist skills to build those controls.

An autonomous actor the usual controls don't reach

An agent is best understood as an autonomous software actor: software that acts on its own, on behalf of the organisation. That is true whether a person starts the agent or a schedule does. Once started, the agent chooses its own steps, as Meta's agent did when an engineer asked it for help. The Open Worldwide Application Security Project (OWASP) publishes the security community's list of top risks for AI applications. One of them is "excessive agency". It happens when an agent is given more power than its task needs, and something then misuses that power. That something could be the model's own mistakes, instructions hidden by an attacker, or a compromised tool or another agent.

Agents often get this extra power by accident. They are run on staff or shared accounts and inherit everything those accounts can do. Only 22% of organisations in one survey give agents their own identity. Extra power also builds up in day-to-day work. Broad access makes an agent "just work". A tool connected for one job often brings many other functions with it. And people click "always allow" to stop the approval prompts.

What the agent is given. OWASP names three common mistakes:

giving an agent tools that can do more than its job needs, such as an email tool that can also delete messages;
giving it more access to systems and data than it needs, such as permission to change and delete records when it only needs to read them;
letting it take big actions without a person checking first.

What attackers feed the agent. An agent reads documents, emails and web pages. It can treat instructions hidden in them as if they came from its user. This is called prompt injection. OWASP gives an example. Take an agent that summarises emails but can also send them. An attacker sends it an email with hidden instructions, and the agent forwards private emails to the attacker.

The US National Institute of Standards and Technology (NIST) tested attacks like this on one agent. The strongest new attack it developed succeeded 81% of the time. The strongest existing attack succeeded only 11% of the time.

Too much power and hidden instructions together are what make an agent dangerous. An attacker's hidden instruction can only do as much harm as the access the agent was given. And the controls we rely on for people don't work the same way on an agent:

Instructions don't bind it. The Medicare agent was blocked, and it found a way around the block.
It is non-deterministic. Normal software follows the same steps every time. An agent works out each step fresh, so the same request can lead to different actions. In the same tests, attacks succeeded 57% of the time on average with a single try. Given 25 tries, they succeeded 80% of the time.
It doesn't stay the same. When the model behind an agent is updated, the agent's behaviour can change even if nothing else does. Researchers found the same AI service behaved quite differently a few months apart, including in how well it followed instructions. So a control that held last month may not hold today.
Legal accountability stays with people and organisations. In a 2026 case about Perplexity's AI agent, a United States appeals court described it as "a tool, not a person" under the law in question.

So the controls have to be built into the architecture. Then they have to be implemented, watched in operation and reassessed regularly, because models, tools and attacks change fast. And a more capable agent does not mean a more ready organisation. It is simply better at finding a way around a block.

Deciding how much authority an agent gets

The controls an agent needs are not new. OpenAI published most of them in 2023, and OWASP recommends the same. The hard part is building them and proving they work.

1. Set autonomy for each action, by its consequence. Describe what the agent is for and what each of its actions does. Note which data and systems it can reach, and what happens if each action goes wrong. Then decide the autonomy for each action. Autonomy is a design decision, separate from what the agent is capable of. The same agent can work alone on low-risk tasks and wait for a person on high-risk ones. Keep each agent modular. Fewer tools and actions mean fewer decisions to set, less access to grant and less to test.

This may seem hard, but it is feasible. Singapore's agentic AI framework describes an IT help-desk agent. Each type of ticket it handled was "assessed for severity of impact, reversibility and feasibility of human oversight". This gave three levels, from fully automatic to fully blocked.

This classification decides which actions need approval. Each action runs in one of three modes: fully autonomous; human-on-the-loop, where a person monitors the agent and can stop it; or human-in-the-loop, where a person approves the action before it runs. Not every action should need approval, because neither form of human oversight is effective by default. Anthropic found that users approve 93% of permission prompts, which leads to "approval fatigue". And no one can monitor thousands of actions. So keep human-in-the-loop for the few actions that are high-impact or hard to undo. Use human-on-the-loop only where actions can be undone, monitoring is automated and raises alerts, and the stop is automatic.

I used the same principle when I built a data-modelling skill with Claude. It studies the requirements on its own, but it does not choose the modelling approach for the information mart layer. It sets out the options (a star schema, a snowflake schema or a data vault), recommends one, and asks the user which to use. The agent does the work, but the important choice stays with a person.

2. Enforce the boundaries outside the agent. Figure 1 shows how this works. The agent can reach enterprise systems only through one request path, and four controls sit around that path. The numbers in the figure match the four controls described below.

Figure 1. The agent reaches enterprise systems only through controls the organisation enforces.

Its own identity, not a borrowed staff account. As noted above, only 22% of organisations do this. When a person starts the agent, it can act as that person instead, but only with the access the task needs, as OWASP recommends.
Automatic checks on every request. An agent makes many separate requests in one session, so a single check at the start is not enough. Each request needs its own check, and these checks run automatically. People approve only the few actions that are hard to undo. The Model Context Protocol is the open standard many agents use to connect to tools. It already requires a token on every request, and accepts each token only at the system it was issued for. It also recommends tokens that expire quickly. Giving each agent an identity of its own is still an emerging area. NIST started work on it in 2026.
Live telemetry and a tamper-resistant audit trail. Telemetry shows what the agent is doing now. The audit trail records what it did. In the same survey, fewer than half of agents were actively monitored. An agent can take thousands of actions in one task, so monitoring has to be automated. Monitor at the control points, such as each request and approval, and watch high-impact actions most closely. The UK AI Security Institute is now adding monitoring "able to flag or block out-of-scope actions as they happen". People see only the alerts, starting with refused requests. Everything else goes to the audit trail.
An automatic stop and a way to undo changes. Monitoring without a stop only records the damage.

3. Test the controls, then roll out in stages. Test first in a closed-off copy of your live systems, and rerun the tests after every model update. But a test environment is never the live system, and the setting changes how agents behave. In one study, changing only the rules around the same agents shifted safety results by 22 to 58 percentage points. So roll out in stages:

start with limited autonomy on a small part of the live work;
watch it with live telemetry and an automatic stop;
pull it back if warning signs appear.

The agent earns more authority through the maturity of your controls, not its own track record. For example, Singapore's framework describes a government agency that built central logging and monitoring before a wider rollout.

Give executives a simple decision

This is where AI governance and architecture meet. The executive who signs off the agent needs a decision on one page. The agent gets the level of autonomy on that page only when every row can be answered with evidence, not intentions. Researchers have proposed something similar: an "autonomy certificate" that "prescribes the maximum level of autonomy at which an agent can operate". The page has five rows:


What the agent can do	Its purpose, and each type of action with its level of autonomy
What can go wrong	The harm each action could cause if the agent gets it wrong, and whether it can be undone
The controls	Which controls apply to each action
The evidence	The latest test results, and when the tests will be run again
The conditions	When the approval ends, what triggers a new review, and which warning signs pause the agent automatically

The last row stops the sign-off from being a one-time event. An agent tested on Monday can behave differently on Tuesday after the vendor updates the model behind it. So the sign-off is not a permanent pass. It is permission to run for a set period while monitoring stays healthy.

A model update, a new tool or a changed system triggers a new review. Autonomy is also withdrawn automatically if warning signs get worse, until someone checks. Warning signs include more attempts to go out of scope, more refused requests and failed scheduled tests.

The business owns this decision.

From my own work

I helped bring in a vendor's agentic solution, delivered as software-as-a-service. Agreeing who was Responsible, Accountable, Consulted and Informed (a RACI) played a vital role. The technology team could be responsible for watching the agent. But accountability stayed with the business, because it chose the solution and used it every day.

This holds true in all but a few cases. Shared platforms, vendor contracts and the type of risk involved can shift part of that accountability. And for an agent built in-house, technology is also accountable for how it is built and updated.

The RACI has to answer one question most miss: who renews the approval when the vendor updates the model? If the contract does not require the vendor to say when the model changes, the business cannot know when its approval has lapsed.

Lessons learned

Start with the action, not the model. Ask what the agent can actually do and what happens if it gets that action wrong.
Autonomy is a business decision, made one action at a time. The same agent can be fully autonomous for one action and require approval for another.
The organisation owns the controls. Do not assume the model's guardrails will protect your systems. Give agents only the access their task needs, and assume anything they read could carry a hidden instruction.
Proven controls earn autonomy. An agent does not earn more authority because it has behaved well. The controls earn it.
Make approval expire. Model updates, new tools, changed systems and warning signs should trigger reassessment.

Sources

OpenAI. (2026, August 26). The Hugging Face incident and the road ahead. Company statement.
Albanese, A. (2026, September 24). Press conference, New York. Prime Minister of Australia. Official transcript.
OpenAI. (2026, September 28; updated October 4). How we will do better for Australia. Company statement.
OpenAI. (2026, July 21). OpenAI and Hugging Face partner to address security incident during model evaluation. Company statement.
AI Security Institute. (2026, August 4). Incident report: Unsanctioned agent behaviour during cyber testing. UK Government incident report.
A rogue AI led to a serious security incident at Meta. (2026, March). The Verge. News report.
Meta is having trouble with rogue AI agents. (2026, March 18). TechCrunch. News report.
OWASP Foundation. (2025). LLM06:2025 Excessive agency. OWASP Top 10 for LLM Applications 2025. Industry standard, not peer-reviewed.
Ruiz, J. (2026, February 4). State of AI agent security 2026 report: When adoption outpaces control. Gravitee. Vendor survey, not peer-reviewed.
Center for AI Standards and Innovation. (2025, January 17). Technical blog: Strengthening AI agent hijacking evaluations. National Institute of Standards and Technology. Government technical report.
Chen, L., Zaharia, M., & Zou, J. (2024). How is ChatGPT's behavior changing over time? Harvard Data Science Review. Peer-reviewed.
Amazon.com Services, LLC v. Perplexity AI, Inc., No. 26-1444 (9th Cir. August 4, 2026). Court opinion.
Shavit, Y., et al. (2023). Practices for governing agentic AI systems. OpenAI. Industry white paper, not peer-reviewed.
Feng, K. J. K., McDonald, D. W., & Zhang, A. X. (2025). Levels of autonomy for AI agents. Knight First Amendment Institute. Preprint.
Infocomm Media Development Authority. (2026, May 20). Updated Model AI Governance Framework for Agentic AI. Singapore Government factsheet. Government framework, not peer-reviewed.
Hughes, J. (2026, March 25). How we built Claude Code auto mode: A safer way to skip permissions. Anthropic. Company engineering blog, not peer-reviewed.
Tallam, K. (2026). Authorization propagation in multi-agent AI systems: Identity governance as infrastructure. Preprint.
Model Context Protocol. (2025). Authorization (Specification version 2025-11-25). Open standard, not peer-reviewed.
National Institute of Standards and Technology, National Cybersecurity Center of Excellence. (2026, February). Accelerating the adoption of software and AI agent identity and authorization. Government concept paper, not peer-reviewed.
Chen, Y. (2026). Institutional red-teaming: Deployment rules, not just models, causally shape multi-agent AI safety. Preprint.

If Tokens Are Getting Cheaper, Why Are AI Agent Bills Going Up?

2026-09-22T00:00:00Z

In short

Underneath, AI runs on tokens, small pieces of text. If you build your own agents, you pay for every token the model reads and writes. If you buy from a vendor, you may be charged per user, per task or per outcome instead, but tokens still drive the vendor's costs, and so your price.

Agents cost far more than chat because they re-read everything, including any document you give them, at every step.

The same quality of AI keeps getting cheaper, but bills have not fallen, because top models cost almost as much as ever and agents use so many more tokens.

Costs are unpredictable, and spending more does not reliably buy better results.

Before approving a budget, ask what a successful result costs and who carries the risk when a task uses more tokens than expected.

I have been curious about token economics for a while. This weekend I finally spent some time, despite the beautiful weather in Melbourne, reading the research, because I wanted to understand what is actually happening across the industry.

The reason was personal. I use an artificial intelligence (AI) coding tool in my day-to-day work. For a long time it ran on a subscription: one flat fee each month. Then it switched to usage-based pricing, where you pay for what you consume. On the first day, I uploaded a medium-sized Word document, the kind most of us write every week, and asked the tool to work with it. Before the day was out I had used up my whole allowance, and the tool stopped working for me.

I had not done anything unusual. So what exactly was I paying for, and why did it run out so fast?

First, what is a token?

Many of you will already know, but it helps to see one. AI models break text into small pieces called tokens. Short words are usually one token; longer words get split, and punctuation counts too. Roughly:

I | uploaded | a | document | and | my | tokens | ran | out | on | day | one | .

That is 12 words and about 13 tokens. The exact split varies by model; you can try it on OpenAI's free tokenizer page. As a rule of thumb, 1,000 words is about 1,300 tokens.

Tokens are the unit that AI model providers charge in, whether you pay them directly or a vendor pays them on your behalf; one recent survey calls them the basic economic unit of agentic AI. But not all tokens cost the same:

Type of token	What it is	How it is usually priced
Input	Everything the model reads	The base rate
Output	Everything the model writes	Several times the input rate
Cached input	Text read recently and reused	Often about a tenth of the input rate
Cache writes	Storing text for reuse (some providers)	A small premium on input
Reasoning	The model's hidden "thinking"	Usually charged as output, though you never see it

Some providers also charge extra for very long inputs or for tool use, such as each web search.

Why agents use so many tokens

A chat is one exchange: the model reads your question once and writes an answer. An agent works through a task in steps. It reads the task, uses a tool (searching files, running a test), reads the result, decides what to do next, and repeats. The catch is that at every step, it re-reads everything that has happened so far.

Every step re-reads everything before it

The pile of text the agent reads grows with every step. Source: Bai et al. 2026.

Put a document in that loop. A 10,000-word document is about 13,000 tokens. A chat reads it once. An agent that takes 30 steps can read it every time: around 390,000 tokens before writing a single useful word. That is an illustration, not a measurement, but it is the shape of what happened to me.

The research backs this up. A University of Michigan and Stanford-led study of eight leading AI models fixing 500 real software bugs found that an agent used about 1,200 times more tokens than a back-and-forth chat about code. Not because agents write more, but because they keep sending the same material back to the model. A Concordia University study of a team of AI agents found that reading made up more than half of all tokens; writing only dominated during the coding itself.

Share of tokens, not share of cost. Source: Salim et al. 2026, Table 3 (30 tasks, one multi-agent system, one model).

Caching helps, but in the Michigan and Stanford study those cheap repeat reads were still the biggest part of the bill. A tiny price multiplied by an enormous volume is still a big number. One caution: nearly all of this evidence comes from coding agents, so your numbers will differ.

If tokens are getting cheaper, why are bills going up?

It depends on what you compare. If you hold quality fixed, AI has become much cheaper: getting the same quality of answer as a few years ago now costs a fraction of what it did. Epoch AI, an independent research group, estimated that this price fell by 9 to 900 times a year up to early 2025, depending on the task. MIT FutureTech, using a stricter method that also counts the extra tokens newer models use, put it at a more modest 5 to 10 times a year. But if you compare the best model then with the best model now, the picture is very different. Here are OpenAI's own published output prices as of September 2026:

Model	Price of 1M output tokens at launch
GPT-4 at launch (March 2023)	$60.00
GPT-6 Astra, newest flagship (September 2026)	$50.00
GPT-5.6 Sol, previous flagship	$30.00
GPT-5.6 Terra, mid-range	$12.00
GPT-5.6 Luna, fast and cheap	$1.20

The cheapest current model writes at one-fiftieth of the 2023 price. But look at the top of the table: the newest flagship costs almost as much as GPT-4 did three and a half years ago, and more than the flagship it replaced. Older levels of quality get dramatically cheaper; the very best stays expensive. So why do bills often go up instead of down?

The best models use far more tokens. Researchers at MIT FutureTech found that while a given level of quality got 5 to 10 times cheaper each year, the cost of running the best available model rose by 3 to 18 times a year, because top models are bigger and do much more reasoning.
Agents multiply every token. A cheaper token read a thousand times is not cheap.
Costs are unpredictable. Running the same agent on the same bug four times, the most expensive run typically cost about double the cheapest, and in extreme cases up to 30 times more. The expensive runs were not better, just full of the agent re-opening the same files. When asked to estimate their own token use in advance, every model came in too low.

Think of it like petrol getting cheaper while you swap a small car for a truck and drive ten times as far. The price per litre falls, but your fuel bill still goes up.

That explains my day one. The tool was almost certainly working in steps, carrying my document along and re-reading it each time. Under a flat subscription I never saw that; under usage pricing it showed up on the first day. It also explains the pricing change: a flat fee works for chat because most people use roughly the same amount, but agents do not.

What to do about it

Design what the agent reads

What an agent reads is a design choice, not a side effect. Decide what it can pull in (a whole file, or just the relevant part?), what it carries forward from step to step, and when it should stop and hand back to a person. This matters even more when several agents work together: in the Concordia study, agents reviewing each other's work used 59.4% of all tokens, while writing the first version used just 8.6%.

Understand how the vendor charges you

If you buy an agentic solution, you may never see a token on the invoice. Vendors charge per seat, per conversation or transaction, per action or credit, per successful outcome (such as a resolved support ticket), or a mix of these with a platform fee.

Underneath every one of these, the vendor is still paying for tokens. The charging model simply decides who carries the risk when a task needs far more tokens than expected. Per token, you carry it. Per outcome, the vendor carries it, and prices that risk into the fee. Bessemer Venture Partners' AI pricing playbook says the same from the seller's side. Pricing is also moving fast: one of the largest business software vendors has changed its agent pricing model three times in about 18 months.

Questions to ask before you approve a budget

What is the billing unit, and what counts as one? Who decides when a task is complete?
Are failed or escalated tasks charged?
What does a successful result cost, including the failed attempts?
What did repeated pilot runs show? Ask for the typical and worst-case cost, not a single estimate.
What happens if our volume doubles, or the vendor changes its pricing model mid-contract?

Lessons learned

The evidence is young and mostly from coding agents, so use it to understand the shape of the problem and your own pilot runs for the numbers. Agents behave less like a flat subscription and more like a utility bill. Budget per successful outcome, expect a range rather than a single number, and design what the agent reads as carefully as what it does.

Sources

Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., Pei, J. (2026). How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. Preprint (version 2, April 2026). arxiv.org/abs/2604.22750 (PDF)
Salim, M., Latendresse, J., Khatoonabadi, S., Shihab, E. (2026). Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering. Preprint, submitted January 2026; not yet formally published, so treat it as unreviewed. Authors describe their findings as preliminary, based on 30 tasks with one framework (ChatDev) and one model. arxiv.org/abs/2601.14470 (PDF)
Chen, Y. et al. (2026). Token Economics for LLM Agents: A Dual-View Study from Computing and Economics. Preprint, survey. arxiv.org/abs/2605.09104 (PDF)
Cottier, B., Snodin, B., Owen, D., Adamczewski, T. (2025). LLM inference prices have fallen rapidly but unequally across tasks. Epoch AI data insight. epoch.ai/data-insights/llm-inference-price-trends
Gundlach, H., Lynch, J., Mertens, M., Thompson, N. (2026). The Price of Progress: Price-Performance and the Future of AI. MIT FutureTech. Preprint (version 2, March 2026); an earlier version appeared at a NeurIPS 2025 workshop. arxiv.org/abs/2511.23455 (PDF)
Bessemer Venture Partners (2026). The AI pricing and monetization playbook. bvp.com/atlas/the-ai-pricing-and-monetization-playbook
OpenAI (2026). GPT-6 Astra: A new generation of intelligence. openai.com/index/gpt-6-astra
OpenAI. API pricing. openai.com/api/pricing (prices change often; checked September 2026)