What makes an AI system agentic
The principal-agent problem
Autonomy levels and oversight design
| Oversight mode | What it means | When appropriate | Governance requirements |
|---|---|---|---|
| Human-in-the-loop | Agent proposes; human approves before any external action is taken. | High-stakes, low-frequency decisions. Legal, financial, clinical actions with significant consequences. Novel or unclear situations. | Approval workflow design, escalation criteria, time constraints on approvals, fallback if human is unavailable. |
| Human-on-the-loop | Agent acts; human can monitor and intervene. Agent pauses for human review at defined checkpoints or when confidence is below threshold. | Medium-stakes, moderate-frequency operations. Routine but consequential work where speed matters and errors are recoverable. | Real-time monitoring interface, intervention mechanism, checkpoint design, confidence threshold calibration, escalation triggers. |
| Fully autonomous | Agent acts; human reviews outcomes retrospectively in audit logs. | Low-stakes, high-frequency operations. Reversible actions. Well-understood, bounded domains with extensive prior testing. | Comprehensive audit trail, automated anomaly detection, retrospective review schedule, circuit breakers for out-of-bounds behaviour. |
Accountability chains
Governance-level threats
Threat
Prompt injection
Malicious content in the agent’s environment (a webpage, a document, a database record) that instructs the agent to take actions outside its authorised scope. The agent cannot distinguish between its legitimate instructions and injected instructions embedded in content it retrieves. Governance response: scope restriction (the agent should not be able to act on instructions from retrieved content that conflict with its system prompt), input validation, and audit trail review for unexpected action patterns.
Threat
Scope creep
An agent that progressively acquires capabilities or access beyond its authorised scope, either through explicit tool calls that weren’t anticipated at design time, or through accumulated context that allows it to construct actions the original scope boundaries didn’t explicitly prohibit. Governance response: least-privilege tool design (every tool the agent has access to should be the minimum necessary), explicit deny-list for tool combinations, and regular scope audits against the original design specification.
Threat
Unverified handoffs
In multi-agent systems, Agent A passes a task to Agent B without validating that the task is within Agent B’s authorised scope, or without sanitising inputs that may contain injected instructions. Governance response: handoff protocols that specify what each agent is authorised to receive, input sanitisation at every handoff boundary, and audit logging of every inter-agent communication.
Audit trail requirements
Agentic AI and the EU AI Act
What organizations must do
- For every agentic AI system: map its action surface completely. Every tool it has access to, every external system it can write to, every API it can call. The action surface is the governance perimeter.
- Classify each agentic system by risk level using the EU AI Act taxonomy if EU-exposed. Agentic systems in Annex III use cases (employment, essential services, credit, law enforcement) are high-risk regardless of their agentic architecture.
- Design the oversight mode per task type, not per system. Document which operations require human-in-the-loop, which human-on-the-loop, and which can be fully autonomous. Enforce those boundaries technically, not just as policy.
- Implement the minimum viable audit trail: every decision point, every tool call, every external action, with timestamps. Retain logs for the period required by applicable law and your post-incident investigation needs.
- For multi-agent systems: define and document the handoff protocol at every agent boundary. Specify what each agent is authorised to receive, require input sanitisation at every handoff, and log every inter-agent communication.
- Apply least-privilege design to tool access: give each agent only the tools it needs for its authorised tasks. Every additional tool is an additional attack surface for scope creep and prompt injection.
- Run adversarial testing before deployment: prompt injection attempts against every tool the agent can access, scope-creep scenarios, and out-of-scope task requests. Test what happens at the boundaries of authorised behaviour.
Common mistakes
Treating agentic AI governance as a technical problem rather than a governance design problem is the most common mistake. The autonomy boundary, the oversight mode, the audit trail design, and the accountability chain are governance decisions that must be made before the system is built, not debugged after deployment.
Assuming that prompt injection is a model problem and not a governance problem is a second mistake. Prompt injection is possible in any agentic system that retrieves content from the environment and uses it as context for decisions. Governance design — scope restriction, input validation, handoff sanitisation — is the primary mitigation, not model hardening.
Conflating human oversight with human approval is a third mistake. Human oversight under Article 14 of the EU AI Act does not require approving every action. It requires that a human with the competence and authority to override can meaningfully do so when needed. Designing for performative oversight — where approvals are rubber-stamped because the volume and speed make review impossible — is non-compliant and creates liability without providing protection.
Deploying multi-agent systems without governance at the system level is a fourth mistake. Agent-level governance (each agent has a well-defined scope) is necessary but not sufficient. The system-level governance — how agents hand off to each other, who is accountable for the overall system outcome, how the system escalates when it encounters a situation none of its agents are authorised to handle — must be designed explicitly.
Open questions
- Whether the EU AI Act will be amended or supplemented to address agentic AI specifically — the Act predates widespread agentic deployment; the Commission has indicated supplemental guidance is possible but no timeline is confirmed.
- How Article 14 human oversight requirements apply to agentic systems that operate faster than human review is feasible — the practical tension between agentic efficiency and Article 14 compliance is unresolved in the statute and not addressed in the AI Office guidance published to date.
- Whether multi-agent systems where each agent is developed by a different provider create gaps in the EU AI Act actor-role framework — the Act defines roles for individual systems; the accountability structure for emergent multi-system behaviour is not explicitly addressed.
- How prompt injection liability is allocated between the model provider (whose model was susceptible), the agent developer (whose tool architecture allowed injection), and the deployer (whose use case created the exposure) — no legal precedent established as of August 2026.
Frequently asked
FAQ
FAQ
FAQ