A vendor sends over a slick AI demo and asks for a signature. Somewhere behind that demo sits a model nobody outside the vendor's own team has properly interrogated: what it was trained on, what it does badly, who is accountable when it gets something wrong. A model card is the document meant to answer those questions before anyone signs anything.
We build data and AI foundations at Shipshape Data, and model cards come up constantly, both when clients are evaluating vendors and when they are documenting their own models for the first time. This guide covers what a model card actually contains, why the format has stuck since 2018, real examples worth studying, and how to build one without it eating a week you do not have.
What a model card actually is
A model card is a short, structured document that explains how a machine learning model works, what it was built to do, and where it falls short. Think of it as closer to a nutrition label than a sales brochure: it tells you what went into the model, how it performs under different conditions, and the situations where you should not trust its output.
Researchers at Google introduced the idea in a 2018 paper called Model Cards for Model Reporting, and the format has stuck because it solves a real problem. Before model cards, most teams either wrote nothing down or buried the useful detail in a research paper's appendix that nobody outside the lab ever read. A model card puts the essentials in one place: who built it, what data trained it, how it was tested, and what it is not for.
The pattern we see with clients is the same whether they are building models themselves or buying them from a vendor. Teams that write model cards as they go make faster, better-informed calls about where a model can and cannot be trusted. Teams that skip it find out the hard way, usually months later, when the model behaves oddly on a slice of data nobody thought to check.
Why model cards matter for AI projects
Most AI projects that go wrong do not fail because the model was badly built. They fail because nobody understood what they were deploying until it was already live. A model trained and tuned carefully can still be the wrong choice for your data, your users or your regulatory environment, and a model card is where that mismatch should show up before it becomes an incident.
They cut risk before you commit
A model card tells you what data trained the model and how it performed across different groups and conditions. That is enough to catch some of the more obvious risks early. A model trained mostly on one demographic, one language, one type of image, is likely to underperform once it meets anything else. Reading the performance section against your own population before deployment, rather than after a complaint lands, is the whole point.
The other thing a model card tells you honestly is scope. A fraud-detection model tuned on credit card transactions was not built to score loan applications, and treating the two as interchangeable because "it's all classification" is how projects go sideways. The intended-use section exists precisely to stop that kind of reuse.
They give governance something to point to
Compliance and legal teams cannot sign off on a model they cannot see into. Model cards give them structured evidence: who built the model, when, on what data, tested how, with what known limitations. When an auditor or a regulator asks how a decision was made, you want an answer that traces back to specific choices rather than a shrug and a promise that the team was careful.
This matters more once several models are running across an organisation. Without a consistent format, every model gets evaluated from scratch, by whoever happens to be asking, with whatever information the original team is willing to dig up. A model card turns that into a repeatable check rather than a fresh investigation every time.
A model that works well in someone else's context will not automatically work in yours, and the model card is where you find that out before deployment, not after.
What goes into a model card
The structure researchers proposed in 2018 still holds up, and most organisations adapt it rather than replace it. You will not need every section for every model, but a card missing all of the following is not really a model card, it is marketing copy in a technical font.
- Model details: who built it, when, what version you are documenting, and what type of architecture it uses
- Intended use: the specific applications it was designed for, and what falls outside that scope
- Training data: where it came from, how it was collected, and what preprocessing was applied
- Quantitative performance: accuracy, precision, recall and other relevant metrics, broken down by group or condition
- Ethical considerations: fairness concerns, privacy implications, likely misuse
- Recommendations: how the model should be monitored, retested and revalidated as it meets new data
The identification basics
Start with model details covering the basics: who built it, when, what version this document describes, and what type of model it is. Link to a research paper or technical documentation if one exists, and give a contact route for questions, because "figure it out yourself" is not a governance answer.
Then define intended use properly. List the specific applications the model was designed for, and be explicit about what sits outside that scope. "A quality-control system" tells a future team almost nothing. "Detects surface defects on painted metal panels under fixed lighting conditions" tells them exactly whether it fits their line.
Training data and performance
Document where the training data came from, how it was collected, and what preprocessing happened before the model saw it. You do not have to hand over proprietary pipelines, but you do have to give readers enough to judge where bias might have crept in.
Performance metrics need the same honesty. Report how the model performs across meaningful groups, whether that is demographics, geographies, device types or whatever splits are relevant to how it will actually be used. Include the conditions where it underperforms. A card that only shows the number that looks good is not a model card, it is a highlight reel, and everyone downstream ends up paying for that omission eventually.
Ethical considerations and recommendations
Add a section naming the fairness concerns, privacy implications and wider effects the model might have once it is running. Then close with recommendations: how it should be monitored, retested and revalidated as it meets new contexts over time. This is the part that turns a one-off document into a living record, and it is also the part most cards skip, because it takes actual thought rather than a metadata dump.
How to build one without losing a week
Building a model card sounds like it should take a long afternoon and somehow eats three days instead. The fix is not a better template. It is timing.
Start while you build, not after
Track decisions as you make them, not months later when you are trying to reconstruct why a dataset got dropped or a threshold got changed. Note the training data and why it was chosen, the architecture and any fairness or performance constraints you built in, and save evaluation results the moment tests run, along with the conditions they ran under. A team that documents like this can usually assemble a solid model card in a couple of hours once the model itself is done. A team that waits ends up guessing, and guesses do not hold up under audit.
- Capture the basics as soon as the project starts: who, when, what problem, what version
- Log training data sources and preprocessing decisions the day you make them, not the week before launch
- Save every evaluation result with the conditions it ran under, including the runs that came out worse than hoped
- Write the ethical considerations and recommendations sections last, once you actually know how the model behaves
Use a template rather than starting from a blank page
You do not need to invent the structure. Google's Model Card Toolkit generates a card from your model's metadata and evaluation output. The original 2018 research paper's template is a solid starting point on its own, and cloud platforms increasingly bake the format in directly, Amazon SageMaker's model cards being one example, so documentation becomes part of the pipeline rather than a chore bolted on afterwards. Hugging Face's community model cards show a looser, developer-first version of the same idea, useful if your audience is mostly other engineers rather than a compliance committee.
Pick whichever starting point matches how your organisation works, then adjust the sections to the people who will actually read it. Legal wants more on data provenance. Product wants performance broken out by user segment. Nobody wants a document written for an audience that does not exist.
What good model cards look like in practice
A handful of published examples are worth studying, because they show several genuinely different approaches rather than one correct format.
Meta and Microsoft's Llama 2 card sits inside the research paper's appendix and goes deep: hardware requirements, carbon footprint, a full ethical considerations section. It is written for a technical audience that wants implementation detail alongside the governance material, and it does not apologise for the length.
OpenAI's original GPT-3 card takes the opposite approach. Short sections on model details, intended use, data sources and limitations, plus a direct route for feedback. Nothing decorative, and that restraint is arguably the point: it gets read in full because it takes five minutes.
Google's face detection model card does something neither of the above does. It shows you the failures: blurry faces, crowded scenes, the actual images where the model struggles, not just a paragraph describing the limitation in the abstract. A non-technical stakeholder understands a picture of a failure faster than a sentence about "reduced accuracy under suboptimal conditions".
Hugging Face sits at the other end of the scale entirely. Its community model cards are short, developer-written, and there are hundreds of thousands of them, because publishing one is close to a condition of sharing a model on the platform at all. Individually they are not always polished, and plenty are thin. Collectively they show what happens when documentation becomes a norm rather than a special project: it gets easier and more consistent over time, not harder, because nobody is starting from a blank page.
The lesson across all four is the same: match the format to the reader, not to a template you found once and now copy without thinking. A card written for engineers and handed to a compliance officer helps nobody, and the reverse is just as true.
Common mistakes that make a model card useless
We have read plenty of model cards that exist purely to tick a compliance box, and they tend to share the same faults.
- Performance reported as one headline number, with no breakdown by group, region or condition where it actually matters
- Intended use written loosely after the model was already deployed for something else, so the card describes whatever the model happens to be doing rather than what it was built for
- Training data description vague enough to hide exactly the bias questions the card exists to answer
- No update after retraining, so the card describes a model that stopped existing eighteen months ago
- Buried in a wiki or a shared drive that nobody outside the original team knows to check
Fix the last one first, even though it is the most boring. A brilliant model card nobody can find might as well not exist.
Evaluating vendors with model cards
The moment a vendor puts an AI product in front of you, ask for the model card before the contract, not after. A serious model card tells you whether the vendor trained on data that resembles yours, whether they tested across the groups that matter to your business, and what they already know is wrong with it.
We have sat through enough vendor pitches to know what it means when someone gets vague at this point. "We use industry-standard datasets" is not an answer. Neither is a performance number with no breakdown behind it. If a vendor cannot produce a model card, or produces one that reads like it was written by legal rather than the people who built the model, treat that as information in itself.
Comparing vendors side by side gets much easier once you are reading the same document type for each. You can line up training data provenance, tested demographics and known failure modes across three shortlisted tools in an afternoon, instead of taking each vendor's word for it separately and hoping the gaps line up in your favour.
A few questions tend to separate a genuine model card from a document written to pass a checklist. Does it name the actual datasets, or just describe them as "diverse"? Does the performance section break out results by anything other than the single number the sales deck already showed you? Is there a date on it, and does that date predate the version of the model you are actually being sold? If the answer to any of those is no, ask again before you sign, not after.
Making model cards part of how your organisation works
The organisations that get real value from model cards are not the ones that write one occasionally when an auditor asks. They are the ones that require a card for every model before it ships, whether it was built in-house or bought in.
This is the part of AI governance we spend a good deal of our time on with clients: not choosing tools, but building the habit of documenting decisions as they happen, so the organisation has an honest record to work from later. Getting validation and testing woven into that record from the start makes the recommendations section something people can actually act on, rather than a paragraph of good intentions. A standard template that legal, compliance and product all agree on before the first model ships saves the argument about what "good enough" documentation looks like every single time a new model turns up.
Get the documentation habit right, then everything downstream is easier
Model cards will not make a mediocre model good. What they do is stop you finding out it is mediocre at the worst possible moment: in production, in front of a customer or a regulator. Treat them as part of building the model rather than paperwork tacked on at the end, and the rest of your AI governance gets noticeably less painful.
If you want help setting a model card standard across your AI systems, or getting your data into a state where documenting a model does not mean chasing three teams for information nobody wrote down, talk to us. We help organisations build the kind of data foundation that makes questions like "what is this model actually doing" easy to answer.