Spotlight on agentic AI | AI governance framework and oversight

AI Governance frameworks designed for generative AI largely focus on output review: a human evaluates what the system produces before anything happens as a result. Agentic AI requires a fundamentally different approach.

Where a system can access data, make decisions and initiate actions, governance cannot focus solely on outputs. It has to address what the system is doing throughout the workflow. This changes what governance frameworks need to cover, how oversight is designed and what organisations need to be able to prove afterwards.

This article sets out the governance essentials for organisations deploying agentic AI.

Determine the use case(s)

At the outset, organisations should make a risk-based determination to decide which use cases are suitable for agent development and/or deployment. For example, an agent with write access to sensitive customer data will have greater risk than one that creates summaries of internal meetings. 

Define the permission boundary

Once use cases have been selected, organisations should define which systems, databases and data sources the agent can reach, which actions it can take without human approval, and what it is expressly prohibited from doing. Depending on the type of deployment, managing or controlling the token consumption costs incurred by the agent may also be an important factor. 

In some deployments, the legitimate set of legal acts (if any) that the agent may take on the organisation’s behalf will be narrow. Authority to act should be a deliberate design choice, rather than a default that follows from giving the agent system access.

As noted in our Understanding Agentic AI article, agency and autonomy do not always move together. A system may have broad access but operate under tight instructions, or limited access but wide discretion. Both dimensions of AI autonomy need to be defined and controlled.

Permissions described in the contract should be reflected in the system's operational configuration. A contractual permission boundary that is not enforced in practice offers limited protection.

It is also worth establishing accountability mapping at the outset, for example through a RACI matrix that assigns ownership for model choice, system settings, monitoring and incident response. Responsibilities should be allocated appropriately across the organisation. For example, a cybersecurity SME should be defining and testing the agent’s security guardrails, rather than a product manager. Where something goes wrong, the question of who was responsible for what should not need to be worked out from scratch. 

Design oversight that works in practice

As agentic systems may take consequential steps at speed, AI oversight needs to be built into the workflow rather than applied after the fact. The three principal models are:

• Human-in-command: the system operates under close human direction, with strict limits on autonomous action. This is appropriate where the risk of unsupervised action is high.
• Human-in-the-loop: defined checkpoints at which human approval is required before the system can proceed, triggered by types of action, and/or by specified thresholds such as transaction value, counterparty risk or confidence score.
• Human-on-the-loop: the system operates within defined parameters, subject to continuous monitoring and the ability to intervene.

Approval gates need to be realistic in operation. A control that requires human sign-off within seconds in a high-volume workflow is unlikely to be meaningful. At greater scale, human-in-the-loop oversight may become operationally unviable altogether.

In higher-autonomy deployments, useful guardrails may also include hard limits on time spent, cost incurred or transactions executed in a single workflow – so that the system stops or escalates before an error has a chance to compound.

Where internal teams are building agentic workflows, there may also be an opportunity to embed compliance by design, building mandatory compliance checks directly into the agent's operations and tool calls, rather than relying solely on downstream review. 

Build the audit trail from the outset

A central governance question for any agentic deployment is what the organisation will be able to prove afterwards: what was accessed, what actions were taken, and on what basis.

Organisations should ensure the system generates logs at a level of granularity sufficient to reconstruct what happened across the full workflow, that those logs are accessible in a usable format, and that they are retained long enough to support regulatory enquiries, third-party claims and internal accountability.

Where possible, the framework should require the system to record not only what it did, but the inputs or reasoning on which it relied and the outcome it was seeking to achieve. In multi-provider deployments, the question of where logs sit and how they can be assembled into a coherent evidence trail needs to be addressed explicitly, both in the governance framework and in the underlying contracts.

In particular, write or execute steps may benefit from a contemporaneous ‘action justification’ generated by the agent itself – capturing, at the point of acting, why the step was taken and what outcome was expected. Recording the basis for the action in the moment is more reliable than reconstructing it after the fact.

Where this evidence is incomplete or fragmented, it may be harder to justify continuing use of the agent to regulators. The audit trail therefore functions not only as an incident-response tool, but as a precondition of safe deployment.

Monitor continuously

As with AI more broadly, governance is not a set-and-forget exercise. Agentic systems evolve, platforms change and the regulatory landscape moves. A framework that was adequate at deployment may not remain so.

Organisations should monitor whether the system remains within its permission boundary, whether approval gates are operating as intended, and whether changes to tools, configuration or the vendor stack alter the risk profile.

The framework should specify what is monitored, at what frequency, by whom, and what triggers a formal review or escalation. 

Know what to do when something goes wrong

With agentic AI, harm may already have occurred by the time an issue is detected. An unauthorised transaction may have completed, misleading communications may have been sent, or data may have been accessed inappropriately. Identifying responsibility may also require untangling actions across a fragmented vendor stack.

AI Governance frameworks should address how the organisation will detect that something has gone wrong, how the system can be suspended quickly if necessary, and how it will reconstruct events and meet any notification or reporting obligations to regulators, affected third parties and, where relevant, data subjects.

The right to suspend agentic functionality quickly, and the supplier's obligation to support that, should be reflected in both the governance framework and the underlying contract. A governance right without contractual backing may be difficult to exercise in practice.

Governance and contracting as an integrated framework

For agentic AI, governance cannot be bolted on later, and it cannot be separated from contracting. The permission boundary, oversight model, audit requirements and incident response procedures all need to be reflected in both the legal and the operational architecture.

The contracting article in this series addresses the contractual framework that sits alongside these governance essentials. The two need to be designed together, not in sequence.