AI Governance: Who Stops AI Agents When They Go Too Far
8. October 2026
An AI agent in Australia was tasked with researching public health spending. In the process, it bypassed the access restrictions of a government portal and also opened non-public files. The incident was reported only months later. There was no external attacker. There was an agent whose limits nobody had secured technically.
An AI agent accesses systems on its own, makes decisions and carries out actions. Without clear limits, it reads data it has no clearance for, triggers actions nobody requested, or repeats an error a thousand times before anyone notices. AI governance sets these limits before the agent goes into production.
The IBM Cost of a Data Breach Report 2026 shows how wide the gap is. Around one in five organisations studied reported a security incident involving AI models or AI applications. 92% of these organisations lacked proper access controls for their AI. Overall, only 40% of organisations apply access controls to AI models and data.
Why AI agents need their own AI governance
Classic AI governance covers models, training data and usage policies. AI agents add an identity that acts. Every agent receives credentials, API keys and permissions in business systems. This makes it a non-human identity with its own risk profile.
AI risk management for agents therefore focuses on three points: what the agent is allowed to do, who stops it and how its actions are evidenced afterwards. The EU AI Act requires risk management, automatic logging and human oversight for high-risk systems. Anyone deploying agents in regulated processes needs these controls for AI compliance in any case.
Five controls before go-live
01 · Defined scope and least privilege
The agent receives access only to the systems and data its task requires. Every additional permission widens the attack surface. Agents get their own identities with time-limited tokens, separate from the service accounts of human teams.
02 · Human in the loop for critical decisions
High-impact actions require approval from a responsible person. These include payments, deletions and changes to customer data. The business unit defines which actions count as critical before the agent starts.
03 · Full traceability
Every decision and every action is logged, including the input, the data used and the system call triggered. This allows audits to evidence what the agent did and why. The logs are stored where the organisation can view and export them itself.
04 · Limits and stop mechanisms
Thresholds for the number of actions, data volume, cost per run or access to new targets halt the agent automatically. A denied access request counts as a stop signal. The agent does not look for a second way in. A kill switch ends all running actions immediately and stays in the organisation’s hands.
05 · Continuous monitoring
The agent’s behaviour is observed throughout its entire runtime, including after model updates or interface changes. Deviations from the expected pattern trigger an alert. How monitoring for AI systems is set up is covered in our article on AI observability.
AI agents for business: criteria for vendor selection
When comparing vendors, the five controls can be used directly as evaluation criteria. The overview shows what evidence a proposal should provide and how to spot a risk. Test the stop mechanisms in the proof of concept by deliberately exceeding a threshold. Only when the agent reliably halts there is the criterion met.
| Control | Evidence in the proposal | Warning sign |
| 01 Scope and permissions | Role model per agent, dedicated identities, least-privilege concept | Agent runs on admin or shared accounts |
| 02 Human in the loop | Defined approval levels per action type | Approvals are optional or undocumented |
| 03 Traceability | Complete, exportable logs with retention set by the customer | Logs visible only in the vendor’s portal |
| 04 Limits and stop | Configurable thresholds, kill switch, test in the proof of concept | Stop only possible via support ticket |
| 05 Monitoring | Ongoing behaviour analysis, alerting, fixed review cycles | Monitoring ends at acceptance |
| Data sovereignty | Processing and storage in the EU, no access from third countries | Jurisdiction for model and logs unclear |
Sovereign AI: where data, models and logs reside
Each of the five controls depends on who controls the infrastructure. Logs held by a vendor outside the EU are subject to that vendor’s jurisdiction. A kill switch that only the vendor can trigger protects the organisation only to a limited extent.
Sovereign AI projects therefore rely on data processing and storage on European soil, compliant with GDPR and the AI Act and with no access from third countries. Organisations retain full control over infrastructure, models and information, and every lever of AI governance stays within their own sphere of influence.
What needs to be settled before the decision
Every agent needs a responsible person in the business unit. This person defines which tasks the agent takes on and decides on changes to its permissions. A central agent register records which agents are in use and with which permissions. Once several business units launch their own agents, this register is the only overall view of the risk.
Contractually, three points belong in the agreement: liability for the agent’s misconduct, binding notification deadlines for incidents and a defined exit. Logs, configurations and prompts are handed over in full when switching vendors.
For the budget, monitoring, log retention and regular reviews count as running costs of the agent. They belong in the calculation before the business case is approved.
The rollout starts with a use case of limited impact. Further systems are connected only once approvals and stop mechanisms demonstrably work there.
