AI governance: a practical roadmap

AI governance has moved quickly from an emerging issue to a practical priority for organisations developing, procuring or deploying AI. Effective governance is not a one-off policy exercise: it needs to evolve as systems change, regulation develops, and organisations learn from their own deployments.

This article sets out a practical framework for building or improving an organisation’s AI governance programme, drawing on the three-phase approach Vik Khurana and Simon McDougall explored in The Roadmap podcast series: identify and assess, design and implement, and manage and monitor.

Phase one: identify and assess

You cannot govern what you cannot see. The starting point for any AI governance programme is building an accurate picture of how AI is being used across the business.

This can be trickier than it sounds. AI may be embedded in HR tools, finance platforms, customer service software, analytics tools, productivity suites and supplier services. A structured discovery exercise - involving IT, procurement and business unit leads - can be useful before any meaningful risk assessment begins.

Practical considerations at this stage may include:

• Issuing a structured questionnaire to business units asking them to identify AI tools in use, the decisions those tools support, and the data inputs involved.
• Mapping identified uses against a risk classification framework. The EU AI Act’s prohibited and high-risk categories may provide a useful reference point, even for organisations not directly subject to the Act, because they reflect considered regulatory thinking about where AI may cause harm.
• Identifying the organisation’s role in respect of each system: is the organisation a provider, a deployer, or both? The answer may affect which obligations attach to the organisation and which sit with a supplier.

Phase two: design and implement

Once an organisation has a baseline picture of AI use, the next step is designing governance structures that are proportionate to its risk profile.

A common mistake is over-engineering the framework at the outset. Governance that is too complex to operate becomes shelfware. A better approach is often to start with a minimum viable framework - clear ownership, a risk classification tool, and a defined process for approving new AI use cases - and build from there.

Practical considerations at this stage may include:

• Assigning clear ownership. AI governance needs technical input to assess system behaviour and legal or compliance input to assess regulatory exposure.
• Building a requirements inventory: a living document that maps regulatory obligations, such as the EU AI Act, data protection law and sector-specific rules, to specific governance controls.
• Conducting gap analysis against a recognised framework. ISO/IEC 42001, which addresses AI management systems, and the NIST AI Risk Management Framework are both useful practical references.
• Integrating AI review into existing procurement and change management processes rather than creating a parallel track. 

Phase three: manage and monitor

Governance fails when it is treated as a one-time exercise. AI systems change: models are updated, data inputs shift, and the decisions or processes they support may evolve.

Building AI governance into business-as-usual processes is the defining challenge of this phase. That means two things in practice: ongoing monitoring of system performance and building a culture of AI literacy across the organisation.

Practical considerations at this stage may include:

• Defining key performance indicators for each material AI system before deployment, not after. What does good look like? What would trigger a review, escalation or suspension?
• Establishing a regular review cadence - for example, more frequent reviews for higher-risk systems and lighter-touch reviews for lower-risk deployments.. Reviews should consider both technical performance and any changes in the regulatory environment.
• Investing in AI literacy training tailored to role. Board members need a different briefing to data scientists. Legal and compliance teams need to understand enough about how AI systems work to spot the questions they should be asking.
• Documenting governance decisions. If a regulator, customer or affected individual later asks how an AI system was approved, monitored or reviewed, the answer needs to be in writing.

A note on agentic AI

Many organisations’ first-generation AI governance programmes were developed with generative AI assistants in mind. Agentic AI - systems that can take actions, use tools and operate with greater autonomy - raises additional questions that conventional governance programmes may not fully address.

For example, organisations may need to consider how agentic systems are authorised, what tools they can access, what actions they can take, how those actions are logged, when human approval is required, and how unexpected behaviour is escalated or stopped.

We explore this further as part of our agentic AI series – see here.

Listening further

Vik Khurana and Simon McDougall explore each of these phases in depth in The Roadmap podcast series, including practical perspectives from Simon’s experience as Chief Strategist for Privacy and AI at ZoomInfo. The episodes are available below:

• Identify & Assess
• Design & Implement
• Manage & Monitor
• Top tips from a technology executive