Data pipeline documentation: a template and worked example

A pipeline fails early on a working day. The engineer who built it left months ago. The person on call opens the repository and finds code and a scheduler entry. Nothing says what the job is for, which reports depend on it, or whether it is safe to rerun. Most teams go through this once before they decide that pipeline documentation matters.

Data pipeline documentation is the written record that explains what a pipeline is for, where its data comes from, what it assumes, what limits it works within, who owns it, and what to do when it fails. Good documentation lets someone who did not build the pipeline run, fix and change it safely.

This guide is for data engineers, analytics engineers and data leads who own pipelines that feed real decisions. It gives you a template with ten sections, a filled example for a fictional but realistic pipeline, and a plain outline you can copy into your own repository. It also covers how to keep the document current once the first version is written, which is where most documentation fails.

Shipshape Data builds data foundations and the artificial intelligence that uses them. We see a consistent pattern across many organisations. Teams are capable and code functions, yet the purpose and assumptions of data pipelines are frequently known only to the individuals who created them. The following template represents the documentation we recommend for every pipeline that supports business decisions.

Why documentation must outlive its author

Code tells you what a pipeline does. It rarely tells you why. A join on a particular key, a filter that drops some rows, a hard-coded date: each one was a decision, and the reason for it is not in the code. When the author moves on, the reason goes with them. The next person either leaves the logic alone because they are afraid to touch it, or changes it and breaks something they could not see.

This situation generates costs in three areas. Incidents take longer to resolve because the on-call person must identify what normal behaviour looks like before they can fix the problem. Making changes becomes risky because it is unclear which reports, models, or teams rely on the data output. Answering audit and regulatory questions is hard because there is no documentation showing how a number moved from its source to a board report. Our guide to data pipeline architecture explains how to design pipelines correctly. This guide explains how to document that design so the information stays available after the original creators leave.

Code demonstrates the function of a pipeline. Documentation describes the pipeline's purpose and its dependencies.

Documentation does not need to be long. It needs to answer the questions a stranger will ask at the worst possible moment: what is this for, what does it trust, who do I call, and is it safe to run again.

A guide to each section of the template

The template contains ten sections. Each section addresses one specific question. You should keep these sections concise. Provide links to the code, tests, or lineage graph for more detail rather than copying that information into the text.

Purpose and the decision it serves

Write one or two sentences describing the output of the pipeline and the specific decision or process that relies on it. Identify the report, model, or system that uses this output. If no one can identify a specific decision served by the pipeline, document this as a finding; the pipeline might be unnecessary.

Sources

List every source system, table, file and API. For each one, record the access route, the extract method (full load, incremental or change data capture), how often the source refreshes, and who owns it. Note any source that sits outside your organisation, because those change without warning.

Assumptions

Document the assumptions the pipeline makes about its inputs that are not strictly enforced. For every assumption, describe how it is verified or state clearly if no check exists. The following section of this guide provides more information on assumptions.

Constraints

Record the particular limits that impact the pipeline. The list comprises load windows, API rate limits, storage regions, and retention rules. Also include cost ceilings and any legal or contractual restrictions. These constraints give the context for design decisions that could otherwise appear unusual.

Transformations

Outline the primary steps using business terminology: filters, joins, deduplication, currency conversion and business rules. Provide a link to the code for the details. When a step implements a business definition, for example what qualifies as an active customer, refer to the agreed definition in your data dictionary instead of redefining it.

Owners

Specify a business owner, a technical owner, and a deputy. Indicate the owner for each source. Give a precise role and a named individual for each position. A team name does not count as a valid owner.

Service levels

Document the specific promises made to the people who use the output. List the deadlines for data delivery, the required level of completeness, the specific quality checks that must pass, and the hours when support is available for failures. If you have not made any formal promises, record that fact, as users will otherwise make their own assumptions.

Failure modes and runbook

This is the pipeline runbook. List the known ways the pipeline fails. For each one, record how it shows up (an alert or a symptom), the first diagnostic step, the fix, whether the job is safe to rerun, how to backfill missing dates, and who must be told.

Lineage

Identify the sources and destinations of the data, both upstream and downstream. Provide links to any lineage graphs created by your tools. Document any manual steps that the tools do not track, such as when a team exports data to a spreadsheet. Our article on what data lineage is and why it matters explains the broader practice in detail.

Change log

Document every significant change by recording the date, the specific modification, the reason for the update, the person who approved it, and the users who were notified. This change log provides the historical context a future owner requires to understand the current structure of the pipeline.

Assumptions, constraints, and owners

Most pipeline documentation describes sources and transformations clearly because these elements are located near the code. The three sections that authors omit most frequently are assumptions, constraints, and owners. These three categories are the most important pieces of information to have when a system fails.

Assumptions are beliefs you do not enforce

An assumption is something the pipeline relies on being true but does not check, or checks only loosely. Examples include a source sending one row per order, a timestamp updating on every change, or a file always arriving before a given time. Assumptions are dangerous because they are invisible in the code and true until the day they are not.

Each written assumption should lead somewhere. The best outcome is a test that fails loudly when the assumption breaks. The next best is an agreement with the source owner, recorded as a data contract, so that a change at source is announced before it arrives. The minimum is an honest note that the assumption is unchecked, with a named person who can confirm it.

Constraints are limits that you cannot change

A constraint is a fixed limit set by something outside the pipeline: a system that can only be read overnight, an API plan with a request limit, a policy that personal data must stay in a UK region, or a finance rule about how a figure must be calculated. Constraints explain why the pipeline was built the way it was. Without them, a well-meaning engineer may simplify the design and break a rule nobody told them about.

Owners are individuals rather than groups

The business owner defines the purpose of the output and approves any changes to it. The technical owner is responsible for maintaining the pipeline and the integrity of its code. Source owners are the points of contact for changes to input data. You must document the name of each owner and designate a deputy for the technical owner. Update this documentation whenever a staff member changes roles. If ownership is not explicitly recorded, responsibility often falls by default to the person who most recently modified the code.

If you do not document an assumption, you are waiting for a failure to occur.

A practical example: daily order margin

The example below is for a fictional wholesale distributor. The pipeline and the people in it are illustrative. It shows the level of detail each section needs: enough for a stranger to act, and no more.

• Purpose: produces daily gross margin by product and branch. The commercial director uses it in the morning trading review, and the pricing team uses it for the weekly price review.
• The sources include ERP order lines, which are read from a database replica incrementally based on the last-modified timestamp. The ERP application manager owns this data. Supplier cost prices are provided in a CSV file that the procurement team uploads to a storage bucket every Monday; the procurement lead owns this file. Exchange rates are retrieved from a public rates API.
• Assumptions: an order line does not change once it is invoiced (checked by a test on invoice status). The cost file contains every active product (checked by reconciling against the product master). The last-modified timestamp updates on every change to an order line (unchecked; the ERP application manager has been asked to confirm).
• Constraints: the ERP replica can only be read between 01:00 and 05:00. The rates API plan has a daily request limit. Customer-level data must stay in the UK region under the organisation's data protection policy. Finance policy requires margin to use the cost at the time of sale, not the current cost.
• Transformations: raw extracts land in a bronze layer. The silver layer deduplicates and types order lines, joins cost by product and effective date, and converts values to sterling. The gold layer aggregates margin by product, branch and day, using the gross margin definition in the data dictionary.
• Owners: business owner, Head of Commercial Finance. Technical owner, a named analytics engineer, with a named deputy. Source owners, the ERP application manager and the procurement lead.
• Service levels: the gold table is loaded by 06:30 on working days. If it is late, the technical owner is alerted by 06:45 and the trading review is told the figures are provisional. Failures are supported in working hours; outside them, support is best effort.
• This section describes failure modes and the runbook. If the cost file is missing, margin rows will load without cost data. You should check the bucket, contact the procurement lead, and rerun the silver and gold steps for the specific dates. These reruns are safe because loads replace data by date. If the ERP replica is unavailable, the extract will time out. Do not run the process outside of the read window. Instead, run it the following night and mark the affected day as provisional. If the rates API fails, the pipeline uses the rate from the previous day and flags those rows.
• Lineage: ERP and cost file to bronze, silver and gold, then to the trading dashboard and the pricing model. The pricing team also copies the gold table into a spreadsheet; this copy is uncontrolled and is listed so that changes are announced to them.
• Change log: example entry. Returns are now recorded as negative order lines on the original order instead of in a separate table. The business owner approved this change. Trading review users and the pricing team were notified before the release.

Observe what this example excludes. It does not repeat the SQL or explain the function of the scheduler. Instead, it provides links to those topics and focuses on information that the code itself does not convey. If you need to explain scheduling and retry behaviours, refer to our guide on data orchestration to see what information should be included there.

A data pipeline documentation template to copy

Copy the following outline into a README file in the pipeline folder of your repository. Provide an answer for every prompt. Do not leave any sections blank. If a section is not applicable, state that it does not apply and explain the reason.

• The purpose and the decision served: this includes what the pipeline produces, which decision or process depends on it, and which report, model, or system reads the output.
• Sources: list each system, table, file, or API, with how to access it, how to extract data from it, how often the data is updated, and who is responsible for the source.
• Assumptions: enumerate each belief concerning the inputs, explain how each is validated, or indicate that it is unverified and state who can confirm it.
• Constraints comprise load windows, rate limits, storage regions, retention rules, cost limits, and legal or contractual restrictions.
• Transformations: these are the primary steps expressed in business language. They contain links to the code and to the agreed definitions in the data dictionary.
• Owners are identified as business owner, technical owner, deputy, and source owners. Every owner holds a distinct role and is a named individual.
• Service levels cover data arrival times, completeness and quality checks, and support hours. They also specify the information supplied to consumers when targets are not met.
• Failure modes and runbook: each known failure, how it shows up, first diagnostic step, fix, rerun safety, backfill steps and who to tell.
• Lineage: this comprises upstream sources and downstream consumers, a link to any generated lineage graph, and any steps that the tools cannot detect.
• Change log: date, change, reason, approver, and consumers notified, with the newest entry first.

How to maintain the document

A document created a single time and never revised becomes misleading. That is worse than having no document at all. The following practices aid in preserving accuracy without turning documentation into a separate task that no one can find time for.

Store the file within the repository

Save the document as a markdown file in the same folder as the pipeline code rather than using a wiki. This guarantees that engineers view the documentation each time they modify the code. The document will be versioned together with the code, and the version history will record the specific changes made by each contributor.

Review it with every change

Add a line to your pull request template inquiring whether the change impacts purpose, sources, assumptions, constraints, owners, service levels, or failure modes. If it does, the documentation change should be included in the same pull request and reviewed alongside the code. A change log entry must be included for any change that a consumer would notice.

Generate what you can, then write the rest

Column descriptions, test results, and lineage graphs can frequently be generated from your transformation and orchestration tools. Use them and link to them from the document. Allocate human effort to the parts that no tool can produce: purpose, assumptions, constraints, ownership, and the runbook.

Check it when people move

Check the owners section each time a person joins, leaves, or changes their role. Conduct a scheduled review across all pipelines, perhaps during a standard data governance audit, to identify any missed updates. You must treat a pipeline lacking a named owner as a risk that requires resolution rather than an acceptable condition.

Why this matters more when AI reads the data

Pipelines increasingly feed models and AI features as well as dashboards. A person reading a dashboard may notice a number that looks wrong. A model will use it without question and produce a confident answer built on it. When a pipeline's assumptions break silently, AI amplifies the error rather than catching it.

Documented assumptions, owners and lineage are part of what makes data AI-ready. They let you answer where an AI output came from, what it relied on, and who is responsible for the data underneath it. Without them, an AI feature built on top inherits every unwritten risk in the pipelines below it.

Where this leaves you

Start by documenting the pipelines that directly support important decisions, instead of trying to map every task within the scheduler. Create the ten standard sections for one of these pipelines, using the provided example to set the appropriate level of detail. Prioritise effort on sections covering purpose, assumptions, constraints, owners, and the runbook, since this information is unlikely to be evident from the code alone to someone unfamiliar with the system.

Incorporate the document into the change process. Store it in the repository, review it in the same pull request as the code, and refer to it when staff leave or join. A short document that stays accurate is more useful than a long document that is outdated.

If your pipelines operate but lack clear assumptions or ownership, you have a foundational issue rather than a documentation issue. Our data foundation service builds governed pipelines, medallion architecture, and a semantic layer that includes ownership and lineage records. Our data consultancy service helps teams establish governance and documentation practices for their existing pipelines. Talk to us for an honest assessment of where to begin and whether our services are necessary for your situation.

Frequently asked questions