An AI governance framework is the rulebook for how your organisation builds, ships and watches over AI systems: who signs off on what, which standards apply, how you weigh risk, and what actually stops a biased model or a leaky RAG pipeline from reaching customers. Most organisations do not have one. They have a Slack channel where someone occasionally asks whether legal has seen the new chatbot.
We see the gap constantly at Shipshape Data. A team builds a genuinely useful pilot, gets excited, and ships it before anyone has agreed who owns the model, where the training data came from, or what happens when it gets something wrong in front of a customer. Roughly seventy per cent of AI projects never make it from pilot to production, and unclear accountability is one of the biggest reasons why. This piece walks through building a framework that actually gets used: why it matters, how to design one, the standards worth knowing, the extra care generative AI and RAG systems need, templates you can adapt directly, and how to tell whether the whole thing is working or just adding paperwork.
Why governance breaks down before it starts
Without a framework, AI initiatives run in a grey zone. One team decides a model is fine to ship because nobody told them otherwise. Compliance finds out at go-live, when changing course is expensive and embarrassing. Leadership has no real view of what is actually running against customer or employee data, so every incident lands as a surprise instead of a known risk somebody was already watching.
What a framework actually protects
Governance protects the value you have already built by heading off reputational damage, regulatory fines and the operational failures that quietly erode trust. When the framework documents data lineage, model logic and the reasoning behind a decision, an auditor can verify what happened, and a customer who disputes an automated outcome has something to challenge rather than a shrug.
It also speeds things up, which surprises people who assume governance is purely a brake. Pre-agreed risk tiers and approved patterns mean a low-risk chatbot does not sit in the same queue as a credit-scoring model. Teams stop guessing whether they need sign-off, because the boundary is written down rather than argued out fresh every time.
Look at what happened inside the UK's Department for Business and Trade. Its internal AI governance process took in 28 submissions and sent nearly every one back with changes, most often around data flows that had not been mapped properly and data protection impact assessments that should have existed before anyone wrote a line of code. That is a framework doing its job: catching a problem while it is still cheap to fix, rather than after launch, when the fix becomes an incident report and a compliance meeting.
Scope and risk tiers come before roles
Before you assign a single owner, write down what you are actually governing. List every AI system in use or in planning: the pilots nobody has formally signed off, the production tools everyone relies on, and the third-party AI services already wired into your stack that nobody thinks of as "AI" because a vendor sold it as a feature. For each one, note its purpose, where the data comes from, whether a human sits in the decision loop, and who it actually affects. That inventory alone tends to surprise people. We have sat across the table from operations leads who did not know their customer service platform had quietly switched on an AI drafting feature six months earlier.
Draw the boundary
Decide, in writing, what counts. Most organisations bring in anything that touches customer data, shapes a business outcome, or makes a decision that affects a person. Basic automation, rule-based systems with no learning component, and low-stakes internal productivity tools usually sit outside scope. Write the inclusion and exclusion criteria down in plain language, because vague boundaries are exactly why governance gets ignored: if a team cannot tell in five minutes whether their project needs approval, they will skip the question entirely.
Tier the risk
Not every AI system deserves the same scrutiny, and treating them all the same is how governance turns into bureaucracy nobody respects. Set up three to five tiers based on data sensitivity, how reversible the decision is, regulatory exposure, and the potential for harm. A credit decision engine, a hiring filter or a medical triage tool sits at the top: executive sign-off, real testing, ongoing monitoring. A recommendation widget on your internal wiki sits at the bottom: light documentation, a technical lead's nod, done. Match the effort to the risk, or the framework collapses under its own weight within a year.
Give governance real names, not committees
Vague ownership is where governance frameworks quietly die. "The AI team" is not accountable for anything, because nobody in it believes the buck genuinely stops with them. Name an AI owner for every system, someone who authorises deployment and answers for what it does once it is live. Appoint data stewards who check that source data is what it claims to be and is kept for exactly as long as the retention policy allows, no longer. Put one person in charge of the framework itself: tracking submissions, chasing overdue reviews, and updating the document when the rules underneath it change.
Give each role real authority
Ownership without authority is theatre. An AI owner needs the power to approve low-risk changes alone and a clear line to a review board for anything higher stakes. A data steward needs the standing to pause a deployment over a quality problem, not just flag it and hope someone else acts. Write these escalation paths down. The alternative is the diffusion of responsibility you see in every failed audit, where everyone assumed someone else was checking the thing that broke.
Governance dies the moment "the team" is accountable instead of a person. Name someone.
Build the operating procedures around it
Start with a submission process that captures every governance decision point in one form or workflow: data provenance, model explainability, bias testing, privacy impact, security controls. Route submissions automatically by risk tier and functional area rather than leaving triage to whoever happens to read the inbox first. The Department for Business and Trade's form did something useful here too: it stopped teams duplicating existing data protection impact assessments and pointed them at the right protocol before they had built anything, rather than after.
Document approval criteria and timelines so teams know what "good enough" looks like before they submit, not after a rejection. Specify that high-risk systems need two independent peer reviews, a public transparency notice, and quarterly audits. Set service levels: five working days for low-risk approvals, three weeks for high-risk ones. Cover ongoing governance too, including what triggers a re-review (a major model update, a new data source, a regulatory change) and what happens when a system reaches the end of its life.
The standards worth knowing
You do not need to invent a framework from nothing. A handful of established models cover most of what you need, and layering two or three of them is normal practice, not a sign that you have not decided.
Risk-based regulation: the EU AI Act and the UK's approach
The EU AI Act sorts AI systems into four tiers, unacceptable, high, limited and minimal, and bans some outright, including social scoring and real-time biometric surveillance in public places. Run high-risk AI in healthcare, employment or financial services inside the EU and you face binding obligations on transparency, human oversight and formal conformity checks, backed by fines that reach €35 million or 7% of global turnover. This is not guidance you can treat as optional. You need a process that classifies every use case correctly and gets the required controls in place before anything ships, not after a regulator asks.
The UK has gone a different way. Its pro-innovation approach rests on five principles, fairness, transparency, accountability, safety and contestability, and it deliberately avoids the EU's prescriptive detail, leaving sector regulators to apply the principles to their own patch. That gives you room to move without the EU's compliance load, but you get less concrete guidance on what compliance actually looks like day to day, which means more of the interpretation lands on you.
NIST, OECD and FINOS
The NIST AI Risk Management Framework is the practical option: four functions, govern, map, measure, manage, applied to any AI system regardless of sector. It is popular with US enterprises for a good reason. It hands you playbooks rather than principles, so a team can pick it up and use it on an ordinary Tuesday morning without a consultant in the room. The OECD Principles sit at the other end of the spectrum, high-level and human-centric, adopted in some form by more than forty countries. Reach for OECD when you need to show international alignment, and for NIST or a sector standard when you need to show your working. FINOS is narrower again, built for financial services, with deep requirements around visibility into model behaviour, user interactions and security controls. Most organisations we work with layer these: NIST for structure, OECD for the top-line commitments, EU Act compliance bolted on wherever it actually applies to their business.
Where generative AI and RAG break the old rules
Everything above was written for AI that behaves predictably: same input, same output, testable in the ordinary sense. Generative models and retrieval-augmented systems do not play by that rule, and a framework built for traditional machine learning will miss most of what actually goes wrong with them.
Non-deterministic outputs and prompt injection
You cannot test a generative model exhaustively, because the same prompt can return a different answer tomorrow. Governance has to define acceptable ranges instead of fixed expected results: thresholds for toxicity, requirements for factual grounding, bias indicators that trigger a human review or an automatic block. System prompts and temperature settings need version control too, treated as configuration that requires sign-off like anything else that changes production behaviour.
Then there is prompt injection, where a user's input manipulates the model into ignoring its own instructions or leaking something it should not. Mandate input sanitisation, output filtering, and prompt templates that keep user content separate from system instructions. Be specific about who is allowed to change a base prompt and how you test that change before it reaches production. The Department for Business and Trade example from earlier is instructive here too: nearly every one of its generative AI submissions came back with changes once someone actually looked for the risks specific to how large language models fail, rather than the risks a traditional model checklist was built to catch.
Data boundaries in RAG systems
A RAG system answers questions by pulling from a knowledge base, so the real governance question is what it is allowed to pull from. Define exactly which documents, databases or APIs each system can query, and make sure retrieval respects the access permissions already in place elsewhere in the organisation. A customer support chatbot should never be one badly phrased prompt away from surfacing your internal financial model.
Chunking and embedding choices are governance decisions too, not purely engineering ones. They decide what the system can retrieve and how much context survives the process. Approve which embedding models are allowed to touch sensitive content, decide how documents get split so you keep context without over-retrieving, and attach metadata that supports fine-grained access control. Track lineage from source document through embedding to the generated answer, so when an output looks wrong you can trace it back rather than guess. Set a clear trigger for re-indexing when source data changes, decide who can add a new source to the retrieval corpus, and check periodically that retrieved chunks genuinely support what the model claims, rather than the model quietly filling gaps with something plausible-sounding.
Policies, templates and approval workflows worth stealing
Building every policy from a blank page wastes time you do not have. Borrow structure, then adjust it to your actual risk appetite rather than someone else's.
Policy structure
Start with a model approval policy: who reviews submissions, what documentation they need, how long approval takes. Cover purpose and scope, roles and responsibilities, approval criteria by tier, escalation, and how often you revisit each decision. A high-risk approval typically wants executive sign-off, two independent technical reviews, legal clearance and a quarterly audit. A minimal-risk system needs a technical lead's authorisation and a yearly check, nothing more.
Pair that with a data governance policy written specifically for AI: training data quality, retention, access control, lineage. State that training datasets need provenance documentation and bias testing on record before use. Be explicit about which data classifications AI systems are allowed anywhere near, how you handle synthetic data, and who approves a new data source before it feeds a production model.
Approval workflows that do not stall everything
Route by risk tier rather than making every request join the same queue. A useful shape:
- Low risk: one technical reviewer, three working days, checking basic security and data compliance
- Medium risk: add a legal review and a privacy screening, ten working days
- High risk: a full governance board, security, legal, compliance, technical design authority and business ownership in the room, up to three weeks
Write your decision criteria down so two different reviewers reach the same answer given the same evidence: what explainability a high-risk model needs to show, what bias testing a hiring tool needs versus a content recommender, and when a residual risk is acceptable versus when it blocks deployment outright.
The parts people forget
Monitoring does not stop at deployment. Set alert thresholds for accuracy degradation, prediction drift and input patterns that look wrong, and write an incident response procedure that says who gets told when an AI system produces a harmful output, how fast, and what happens next. Cover version control and change management for model updates, and write down what decommissioning actually looks like, including proper data deletion, well before you need it in a hurry.
Measuring whether it is working, and keeping it alive
A framework you cannot measure is a framework you are guessing about. Track submission volume, approval cycle time by tier, and what proportion of systems get flagged for changes versus waved through first time. Watch the operational signals too: bias incidents caught in production, drift alerts, audit findings. Together these tell you whether governance is catching problems or just producing paperwork after the fact.
The metrics that tell you something real
Process metrics show whether governance helps or gets in the way: average time to approval by tier, rejection rates, and how often teams quietly route around the process altogether, which is the number that should worry you most. High rejection usually means unclear guidance rather than bad proposals. Slow approval usually means you need more reviewers or a simpler workflow, not a stricter one.
Compliance metrics, audit pass rate, regulatory findings, the share of live systems with complete documentation, tell you whether the paper trail would survive contact with a real audit. Risk realisation metrics tell you something sharper: privacy breaches traced to an AI system, bias complaints, performance failures, and what they actually cost. Compare that against what your risk assessments predicted before launch. If the gap is wide, your tiering is wrong somewhere, not just your luck.
Review it on a schedule, or it rots
Put a quarterly review in the calendar: check the metric trends, adjust risk tiers as new threats show up, tighten or loosen approval criteria based on what actually happened rather than what you assumed would happen. Bring the AI owners, data stewards, legal and technical leads into the same room and ask what is creating friction, because the policies teams quietly ignore are telling you something worth hearing. Run a full annual review of the whole structure, folding in regulatory changes, new capability such as agentic systems, and whatever you have learned from other organisations' governance failures, which tend to be more instructive than their successes.
Treat the document as something that changes as you do. The first time you ship a generative AI application, update the templates to cover prompt controls properly. As pilots turn into production systems, adjust the tiers to match what is actually running rather than the theoretical version you wrote eighteen months ago and never revisited.
Where to start on Monday morning
None of this is complicated in principle. It is unglamorous, and it needs someone to actually own it, which is usually the part that gets skipped in favour of writing another policy document nobody reads. Start by mapping what you already have running. Assign real names to real roles. Pick the standards that match your regulatory exposure rather than the ones with the best marketing behind them.
We help organisations design governance frameworks that fit the data infrastructure they already have, rather than a template bolted on afterwards, particularly where generative AI or RAG systems have created risks nobody has quite named yet. If you want a straight read on where your gaps are before you write a single policy, talk to us.