The head of AI at a mid-size logistics company shows a slide to the board. It indicates that the department delivered fourteen pilot projects this year, and that three have been approved for a second phase. He does not mention that the two pilots approved for phase two last year were never implemented.
The data scientist who built the models resigned in March. Her laptop and the files on it were not recovered.
An AI maturity model describes an organisation's progress in using artificial intelligence. The progression begins with informal experiments, then proceeds through consistent pilot programmes, and ends with managed production and large-scale, governed operations. Most self-assessments produce a score that is one stage too high. This occurs because organisations frequently count pilot projects that succeeded once rather than focusing on models that keep running without intervention for several months.
Our company, Shipshape Data, develops data and AI systems. Most of our projects start with a particular kind of conversation. Clients typically describe their operations as being at the second or third stage of a maturity model they saw at a conference. When we look at the systems that actually interact with customers without constant supervision, we often discover that the technology remains in the first stage. These systems are often presented as more advanced than they truly are. This generally does not stem from intentional dishonesty.
They're counting the wrong thing.
This guide outlines the practical stages of AI maturity and offers a method for accurately assessing your organisation. It explains why a gap between public claims and internal reality usually exists. This gap typically arises because companies confuse a successful pilot project with a reliable, permanent capability.
The stages, and what each one looks like
Maturity models are common now. They usually consist of five boxes and an arrow. These models are frequently sold with consulting services that start at the stage the buyer currently occupies. Some frameworks use five stages, while others use four. One of our clients obtained a seven-stage version from a vendor. That vendor offered a tool intended to move a company from stage three to stage five. The particular model is less important than an honest assessment of your current position. This assessment needs more than just reading the labels on the boxes. The following sections describe the actual experience of being in each stage.
Stage one: ad hoc experimentation
One or two technical staff members experiment with different tools. They may build a ChatGPT wrapper or a scikit-learn model using whatever data is available, often without following formal request channels. There is no shared infrastructure or established quality standard. Success depends on whether a demonstration works during a meeting. This phase does not indicate failure. Most organisations start in this way. The most efficient way to progress beyond this stage is to let a few interested people experiment before trying to regulate a process that is not yet established.
The sign for stage one is a missing paper trail rather than a lack of polish. There are no records indicating how many projects were attempted, why certain ones were abandoned, or the data's origin. This approach is acceptable in a small organisation. It becomes a problem when staff in different departments start developing the same project because they are unaware of each other's work.
Stage two: repeatable pilots
The organisation has completed enough projects to form a consistent pattern. There is a standard process for selecting use cases, a preferred cloud platform, and a dedicated data science team of three or four people instead of a single individual. Pilots are developed with a specific sponsor and success metrics defined before the project begins, rather than being created later to fit the results. Most pilots at this stage do not advance. This is a normal and productive part of the process. However, problems arise when there is no record of which projects failed or why they failed. Without this documentation, the same ideas are often proposed again eighteen months later by staff members who were not part of the original project.
Stage three: managed production
Many companies claim they have reached this stage, but few actually have. A model is considered in production when it runs without constant manual supervision. It must have an owner responsible for its operation and a system that alerts someone if performance drops. There must be a clear process for retraining or retiring the model. The data pipeline is monitored rather than assumed to work correctly. The organisation has a specific procedure to follow if a model error affects a customer on a Tuesday morning, instead of having no plan at all.
Stage four: scaled and governed
Multiple production models operate across different business functions. These models use shared infrastructure rather than each team building separate systems. A governance department has the authority to reject use cases based on risk assessments. Model performance is reviewed at regular intervals regardless of whether issues have been reported. Very few organisations have reached this level. We examine companies that claim to be at this stage more closely than those that admit they are not. This stage is characterised by routine operations. It is rarely the subject of presentations because it is no longer treated as a special project but as a standard business process.
Why the pilot count is the wrong thing to measure
Most leadership teams gauge the progress of their AI programmes by the number of pilots they have launched. They might report twelve pilots this year, twenty last year, or forty since the programme began. This metric is easy to report to executives because the total always increases. However, these figures do not indicate actual capability. A pilot and a capability address two different objectives.
A pilot project determines if a process can work for a specific data set while a team monitors it and fixes errors. A capability determines if a process continues to work after the team moves to other projects, the input data changes, and the original setup details are forgotten. These two concepts represent different categories of performance rather than different levels on the same scale.
If you complete one hundred successful pilot projects but fail to integrate them into daily operations, you have not improved your organisational maturity. You have simply repeated the same initial step one hundred times.
The demo that never has to survive the real world
People build pilots using curated data and present them to audiences who want the projects to succeed. These pilots are evaluated based on criteria chosen by the developers. This process is not dishonest. However, a pilot environment is easier than a production environment. The skills needed to create a successful demo differ from the skills needed to maintain model accuracy over six months when using uncleaned data.
A pilot project demonstrates that a model functions in a controlled environment using prepared data. Production testing determines if the model continues to function reliably over time without manual intervention.
The true measure of maturity is the number of pilot programmes that remain operational and unmonitored after one year. It is also defined by whether anyone would notice if one of these programmes stopped functioning. Maturity is not determined by the total number of pilots created.
Counting pilots rewards the wrong behaviour
When a leadership team measures progress by the number of pilots, teams concentrate on raising that count. That response follows logically from their performance metrics. The result is a proliferation of small proofs of concept that are dropped once they cease to be new. Such projects are never evaluated against real customer requirements or tough conditions. A programme with three deployed and monitored models is more mature than a programme with thirty pilots and no active systems.
Why most self-assessments come back a stage too high
When a leadership team fills out a maturity questionnaire, most members assign their organisation a level that is one step higher than an external auditor would. This gap is not solely due to vanity, although vanity contributes. The main cause is a structural issue in how teams answer the questions.
Whoever is in the room decides the score
The people who complete maturity self-assessments are usually the same individuals who advocated for the AI programme. They usually base their ratings on the most successful cases rather than on average performance. One production model that works well is often taken to represent the whole organisation. At the same time, the numerous pilot projects that remain incomplete are overlooked. This occurs because no participant in the meeting is responsible for those failures, and attendees tend to avoid sharing negative information during the workshop.
Ambition gets counted as capability
A simpler mechanism also exists. People answer maturity questions based on their future goals instead of their current status. A team that has agreed to build a model governance function will often score itself as if that function is already active. They feel that their intention is the same as the result. This is incorrect. A model governance function that only exists on a slide has no value when a regulator or a customer asks a difficult question about a model's decision.
We have attended workshops where participants rated their progress as stage three. Twenty minutes later, those same people could not agree on who was responsible for the specific model they used as evidence for that rating. The discrepancy between an initial score and the answers to follow-up questions shows the actual state of a project. You should ask these follow-up questions instead of trusting the initial score.
What production requires that was not asked of a pilot
Most AI programmes stop progressing between stage two and stage three. This delay usually happens for reasons unrelated to the model. A pilot project may achieve high accuracy scores but remain unready for production. This is because the production environment requires a different set of standards and requirements.
Someone has to own it after the excitement fades
Every pilot project has a leader during the development phase. Few projects have a designated owner for the eighteen months following the launch. This lack of oversight causes most projects to fail or to continue operating incorrectly without being noticed. Production maturity requires assigning an owner before the pilot is approved instead of waiting until the project begins to decline.
Monitoring has to exist before anything breaks, not after
A model that works correctly today can become inaccurate in three months even if the code does not change. This occurs because the data used to train the model no longer mirrors the present state of the world. Detecting this data drift before a customer observes it is a reliable method to assess whether an organisation has arrived at stage three. This is also the point where the distinction between a pilot project and a formal model deployment becomes clear. A pilot is usually deployed a single time and then left unattended. A production model is deployed, monitored, and refreshed as the underlying data evolves.
The infrastructure has to be built once, not per project
At stage two, each pilot usually recreates its own plumbing: its own data extraction, its own deployment path, its own method of tracking what changed and when. That works for a single pilot. It quickly becomes costly once there are six of them, each using a slightly different approach to perform the same task. Proper MLOps practice converts six custom builds into a single repeatable pipeline, and lacking it is typically the real cause a "scaling" programme merely runs the same manual process in parallel a dozen times.
How to assess where you honestly stand
Avoid the questionnaire, at least initially, and begin with a list. List each AI initiative the organisation has ever funded on a separate line, noting its intended purpose, whether it is still operating today, who currently owns it, and when it was last verified to work. This single exercise often proves more revealing than any maturity framework, because most organisations find they cannot complete the list. No one maintained it.
Count what is unattended, not what exists
The relevant metric is the number of models that have run autonomously for at least six months without errors. This specifically concerns models that provide outputs to customers or make decisions without human review. This number is precise and transparent. It is usually far lower than the count of pilot projects shown in a corporate presentation.
Ask who gets paged
You should enquire who gets a notification if this model begins delivering incorrect answers in the middle of the night. If no one is tasked with fixing it right away, the model is not truly in production regardless of what the deployment dashboard shows. Proper AI governance aims to formalise this kind of discipline. Lacking such accountability usually indicates that technical maturity has been claimed but not truly established.
The trap of chasing the next stage too soon
Once an organisation acknowledges it is lagging behind its claim, the instinct is to jump forward. Declare a stage four governance function, appoint a head of AI, and purchase a platform that promises to scale everything instantly. We have seen this fail more often than we would like, because governance based on two unstable pilots is theatre rather than capability, and a platform bought to scale AI cannot remedy the fact that the underlying data was never mastered initially.
Organisational maturity works more like physical fitness than a formal qualification. You cannot progress from occasional exercise to running a marathon merely by buying better equipment. Each stage must be fully operational before attempting the next level becomes useful. A stage two organisation that dedicates a year to making its initial pilot projects reliable and autonomous is better positioned than one that uses that year to draft a stage four governance policy that is never enforced. This process needs a data strategy that people truly follow and the patience to guarantee one system works correctly before beginning another.
Often we observe a particular scenario in which a governance function is brought in to oversee a programme that has not been created yet. A head of responsible AI is assigned, drafts a policy document, and sets up a review board. That individual then discovers that only a single model is available for review. The model is a pilot project that has remained inactive after its developer transferred to another team. The policy is helpful, yet it was created at the wrong time. Governance becomes necessary only when there are active projects to oversee. Before then, these administrative structures serve no practical purpose.
Where to start
Compile a list of each model or pilot that has been funded. For each entry, list the current status, the owner, and the date of its most recent review. This document conveys more detail about your real progress than a framework can. You can create it in a single afternoon without engaging consultants.
Choose the pilot project that is closest to full autonomy. Find the one where the gap between "it works in the demo" and "it runs on its own without anyone watching" is minimal. Eliminate this gap completely before starting any new projects. Designate a particular owner for the pilot. Implement monitoring systems for the pilot. Record who will be alerted if the system fails. This constitutes a single production capability that you have achieved. This outcome is more valuable than launching twelve more pilots that stay in the planning phase.
If you want an external assessment of your organisation's current state and the steps needed to reach real production capability, talk to us.
Are you figuring out how this applies to your organisation? Our AI strategy service assists you in pinpointing your first profitable use case and the order for developing the next ones.
Add a readiness score to the brief for prospective firms. Our guide on choosing an AI consultancy explains how to apply this score in a three-week selection process.
Organisations at the experimentation stage can use these checks that predict whether an AI project will stall before committing to the next pilot.
Before moving to the next stage, work through our twenty readiness questions across strategy, people, data and change to see what needs fixing first.
Frequently asked questions
What are the stages of an AI maturity model?
This article outlines four practical development stages. The first stage is ad hoc experimentation, in which one or two people test ideas without shared infrastructure. The second stage involves repeatable pilots that employ a basic process and have a formal sponsor. The third stage is managed production, where a model operates independently under a designated owner and a person handling technical alerts. The final stage is a scaled, governed capability. At this stage, multiple production models share the same infrastructure, and a governance department holds the authority to enforce decisions.
Why is pilot count the wrong way to measure AI maturity?
A pilot project demonstrates that a model works once with selected data while a team monitors it and fixes errors. This process does not reveal whether the model will keep functioning after the team departs or when the input data changes. This article argues that a programme with three models in active use and under monitoring is more advanced than a programme with thirty pilot projects and no active models.
Why do AI maturity self-assessments often score too high?
The individual who completes the assessment is usually the programme champion. That individual often rates the best available example instead of the organisational average. Respondents also tend to describe their future goals rather than their current status. For example, an intention to create a governance function is often reported as if that function is already in operation.
What does production maturity require that a pilot does not?
A designated owner is needed for eighteen months after launch. Having someone support the project during development is insufficient. Monitoring must be established beforehand to detect problems, because a model's accuracy can drop within three months even if the code remains unchanged. Infrastructure should be built once and then reused, rather than rebuilt for each pilot programme; this is what good MLOps practices provide.
How do you honestly assess your organisation's AI maturity stage?
Skip the questionnaire and begin with a list of every AI initiative ever funded: its intended purpose, whether it still runs today, its owner, and the date of the last verification of its operation. The article recommends asking who gets paged if a model begins producing wrong answers at two in the morning; if the truthful answer is nobody, it is not in production in any meaningful sense.