Data and AI systems rely on how data travels from its capture point to its use point. Selecting an incorrect structure for this movement causes all later processes to be delayed. This impact reaches from finance team dashboards to decision-making models.
Data pipeline design patterns are the frameworks that ingest, transform, and deliver data. They comprise batch processing, streaming, ETL, ELT, Lambda, Kappa, and RAG. Each pattern fulfils a distinct purpose. Batch processing fits overnight reporting. Streaming meets particular latency needs, but it adds technical complexity and should be employed only when required.
These structures are referred to as data pipeline design patterns. They serve as the frameworks that ingest, transform, and deliver data. No single pattern fits every task. Batch processing performs well for certain workloads yet fails for others. Streaming solves latency problems, but it creates unnecessary operational complexity if low latency is not needed. At Shipshape Data, we build and modernise the data foundations for AI and analytics. We often see that selecting the right pattern lets a team achieve results within weeks.
Choosing the wrong option stretches a six-week project to six months. The source of this delay frequently stays hidden until the organisation receives the computing cost bill.
This guide outlines seven common patterns used in modern analytics. For each pattern it details how it works, its optimal applications, and the related trade-offs. If you are constructing a new platform, moving to the cloud, or preparing a data environment for AI, concentrate on the two or three patterns that apply to your case and ignore the rest. That is our suggested approach.
1. The AI-ready RAG pipeline
This pattern is gaining traction in 2026 and merges data engineering with AI. A retrieval-augmented generation pipeline, or RAG, handles unstructured content such as PDFs, emails, contracts, and internal documents that have not been indexed. It transforms this content into a format that an AI model can search and use to generate answers.
How it works
A RAG pipeline draws in raw source documents and processes them through a series of stages: extraction, chunking, embedding, and then storage in a vector database. When a user asks the AI a question, the pipeline retrieves the most relevant chunks and supplies them to a large language model as context. The model responds using your material, not based on whatever it happened to learn during training. That distinction is the entire purpose.
Where it fits
This fits any organisation that wants AI rooted in its own knowledge rather than the open internet: internal question-and-answer tools, support assistants, compliance helpers, contract review. It is most effective when the source material is document-heavy and domain-specific, and when a confidently wrong answer would cost you something real.
What you get
- The system ingests documents from various storage locations, including SharePoint, S3, and internal databases.
- Parsing and chunking that divides text into segments while preserving the original meaning.
- An embedding step that converts those chunks into vectors for similarity search.
- A vector store indexed for fast retrieval
- Retrieval and generation, where the store feeds the model the context it needs.
The advantage is that your AI supplies answers based on current, verifiable information instead of fabricated data. However, this method forces you to handle ongoing technical complexities, such as document updates, chunking quality, and embedding drift. You need to consider specific failure types. These include outdated content staying in the vector store, chunk boundaries that split sentences from needed context, and retrieval processes that miss relevant information when users pose vague questions.
The quality of your answers relies on your chunking and embedding strategy. All other components are secondary to this particular choice.
Notes for 2026 teams
Configure automated updates from the start so that documents are re-embedded as soon as source files change. Avoid depending on a fixed schedule that could produce outdated information. Test the retrieval process with real user questions before moving to production. Log retrieval scores and user feedback to drive improvements based on data rather than assumptions.
2. Batch processing
Batch processing is a well-known pattern that remains effective for tasks that do not need real-time updates. This approach gathers data over a defined period and processes it in a single scheduled run. It is predictable, low-cost, and easy to maintain.
How it works
A batch pipeline collects records during a defined interval, such as hourly, nightly, or weekly, and then applies one transformation to the whole set. Cloud services like AWS Glue or Azure Data Factory handle scheduling, data extraction, and loading. This design means you do not have to maintain infrastructure for jobs that run on a recurring basis, for example once per night.
Where it fits
Think end-of-day financial reporting, monthly billing runs, warehouse loads, overnight model retraining. If your team opens the dashboard each morning to view yesterday's data, batch handles that cleanly and there is no benefit in choosing a more complex solution.
What you get
- A three-stage flow: source extraction, then transformation, then load to a destination store.
- A scheduler that initiates each run and manages retries when a failure occurs.
- Recovery that doesn't need someone awake at 3am to babysit it
3. Stream processing
Streaming processes data while it is in motion. It acts on each event as it arrives rather than waiting to collect records for later processing. This technique offers the lowest latency among the options listed. It is useful when a decision requires current information instead of historical data from a previous day.
How it works
A stream pipeline gathers events from a source, modifies them during transit, and sends the result to a destination immediately. Cloud-native services like Amazon Kinesis or Azure Event Hubs manage the message broker layer. These services supply the required infrastructure so you do not need to operate your own broker servers.
Where it fits
Streaming is appropriate where freshness drives the business: fraud detection, live dashboards, IoT sensor monitoring, recommendation engines that must react to what someone just did. If a delay of even a few minutes costs you money or misses the moment, this is the right fit.
We often give this warning to our clients.
Streaming addresses latency issues as a particular solution. It does not outperform batch processing in all cases. When teams adopt streaming without a clear low-latency requirement, they add system complexity without receiving any advantage.
What you get
- Events travel from source producers, pass through a broker, and reach a stream processor.
- A processor either stores results or initiates a downstream action.
- State management for windowed aggregations and joins across concurrent streams
- Sub-second latency when it's built and tuned properly
This shift raises operational requirements. You now need to manage consumer lag, partition rebalancing, and exactly-once semantics. Batch processing did not involve these tasks. Consumer lag should serve as your primary indicator of system health. Design schemas so that updates do not disrupt downstream consumers. Test the system at maximum load prior to deployment. Identify your throughput limits before the system goes into production.
4. ETL
ETL, extract, transform, load, is among the most established patterns in enterprise analytics. It extracts data from source systems, transforms it before reaching the destination, and then loads the cleaned, structured output into a target store such as a warehouse.
How it works
The key characteristic is that transformation logic runs in a dedicated processing layer before data reaches the destination. Your target store receives only clean, validated records. The warehouse lacks raw or inconsistent data that would need later correction, a more difficult task.
Where it fits
ETL suits regulated industries such as finance and healthcare. These sectors need data validated and masked before storage in downstream systems. This approach also handles complex joins across multiple sources and extensive business rules. Using computing resources for these tasks during ingestion is more efficient than handling them later.
What you get
- Three sequential stages exist: extraction from the source, transformation in a processing engine, and loading to the destination.
- All three components can be orchestrated within a single managed service like Azure Data Factory.
- We enforce data quality at load time to keep the warehouse clean by design.
A drawback is that transformation logic resides outside the destination. This hinders rapid updates when business rules change. A typical failure is a transformation bottleneck that lengthens load times as data volumes increase. This eventually leads to an unintended significant problem. You should keep the transformation logic under version control separate from the orchestration code. This guarantees auditable changes and lets you roll back a faulty rule update before downstream users encounter incorrect data.
ETL offers more control than ELT when compliance regulations strictly define permissible data in your warehouse.
5. ELT
ELT, extract, load, transform, reverses the order. It first loads raw data into the destination and then transforms it. This became a sensible default once cloud warehouses acquired enough compute to perform the transformations themselves, eliminating the need for a separate processing layer in the middle.
How it works
Data reaches the destination in its raw format. Transformations are performed within the warehouse using SQL tools. Your team can modify rules without changing the ingestion process. This enables faster iteration when business logic changes.
Where it fits
ELT suits analytics teams that need to update data models frequently and use cloud-based data warehouses. This approach works well when queries are run often and your understanding of the source data is still evolving. These conditions typically apply to most analytics projects in their first year.
What you get
- Extract the data and place it in a raw format within a staging layer inside the warehouse.
- Transformation models operate within the destination and scale on services such as Google BigQuery.
- Analysts adjusting logic without waiting in a queue for engineering support
Rapid iteration is the primary advantage. The challenge lies in governance. Because raw data resides in your warehouse, failing to mask sensitive fields during ingestion will create a significant problem. Failures in this area are often hard to detect. For example, transformation models may keep producing plausible yet incorrect results if the underlying source schema changes. Implement column-level access controls immediately and employ automated tools to track changes in source schemas. It is preferable for a model to fail completely rather than deliver inaccurate data.
6. Lambda architecture
Lambda merges batch and stream processing into a single system to deliver both historical accuracy and real-time responsiveness. This pattern is hard to implement because it needs two parallel processing paths that must be reconciled during queries. The system's complexity resides in this reconciliation process.
How it works
Lambda operates two layers simultaneously. A batch layer reprocesses all historical data on a schedule to produce accurate, complete views. A speed layer processes incoming events in real time to supply data between batch runs. A serving layer merges the outputs from both for any system that queries them. This architecture offers fault tolerance because the batch layer automatically overwrites any errors introduced by the speed layer. This ensures that mistakes persist only until the next batch run completes.
Where it fits
This approach suits workloads that need both historical records and real-time data. Examples include large analytics platforms, fraud detection systems, and recommendation engines where reprocessing the full history corrects the approximations made by the speed layer. It is suitable for organisations that can handle the operational costs of running two systems simultaneously.
What you get
- An immutable master dataset feeding both layers at the same time
- A batch layer producing corrected, complete views on a fixed schedule
- A speed layer writing real-time approximations to a low-latency store
- A serving layer that merges both before results reach anyone
The batch layer delivers long-term accuracy because it self-corrects automatically. The drawback is that engineering requirements double. You must maintain two codebases for similar transformations and conduct additional testing. There is also a risk that the two logic sets diverge and produce different outputs. Share transformation logic between the layers when possible. Instrument each layer independently and create alerts to detect significant differences between their outputs before those results appear in a report.
7. Kappa architecture
Kappa architecture eliminates the batch layer present in Lambda architecture. All data passes through a single stream processing pipeline. This design tackles Lambda's main difficulty, the cost of maintaining two separate codebases that process the same data and must yield identical results.
How it works
Kappa treats each data update as a single stream event. If you need to correct historical results, you replay events from a durable log through the existing pipeline rather than using a separate batch system. This approach delivers low-latency output via a single path and eliminates the need to maintain a second codebase.
Where it fits
Kappa suits teams that already have streaming infrastructure in place and need real-time analytics while avoiding Lambda's high overhead costs. The architecture works best when historical reprocessing is infrequent and the datasets are small enough to be handled via log replay, eliminating the need for a separate batch layer.
What you get
- It consists of a single stream-processing layer backed by a durable, replayable event log.
- A broker layer from services such as Amazon Kinesis
- Historical corrections are handled by the existing pipeline via a replay job instead of a separate process.
- One transformation codebase, so far fewer places for logic to diverge
Its primary benefit is simplicity. Nevertheless, large replays may be slow and expensive. Consequently, Kappa is not suitable for frequent reprocessing or for very large datasets. At the start, you should configure a large event retention window. Doing so avoids the system exhausting its log history during a replay, a predictable failure. Before a real need occurs, test the replay process using realistic data volumes. Document the steps so your team can follow them in an emergency without confusion.
How to choose
There are seven patterns and no single winner, since the best choice depends on your workload rather than a ranking. When feeding unstructured documents to AI, begin with RAG. If a delay of a few hours is acceptable, use batch. If decisions depend on the current moment, choose streaming. If compliance determines what is stored in the warehouse, select ETL. If analysts require rapid iteration on cloud-native models, opt for ELT. If you need both historical depth and live data, use Lambda, or Kappa once your streaming capability is mature enough to handle the load alone.
We see teams committing a particular mistake. They choose a pattern before understanding their own workloads. Selecting the wrong pattern does more than merely slow delivery. It generates technical debt that grows as the data estate expands. First, map your current workloads and requirements to these patterns. Identify where your existing pipelines create bottlenecks or impede AI initiatives. Choose the architecture that matches your current task instead of the one that seemed most impressive at a conference.
If you prefer someone to evaluate the state of your data infrastructure and identify needed changes before you commit, that is what we do. Talk to us and begin with a clear assessment of your pipelines rather than a vendor's slide deck.
Our data consultants develop these systems for UK organisations. We focus on the data foundation to ensure reliable results.
For the software that puts these patterns into practice, read our criteria-led roundup of data pipeline software.
Frequently asked questions
What do data pipeline design patterns have to do with designing data-intensive applications?
Each reliable data-intensive application relies on a set of established pipeline patterns-batch, streaming, ETL, ELT, Lambda, Kappa, and RAG-and each pattern addresses a particular latency or reliability issue rather than serving as a universal upgrade. Choosing a pattern before you understand your actual latency, volume, and reprocessing requirements is the most common cause of a data-intensive system becoming over-engineered or fragile. Streaming, for example, only provides value when a real freshness problem justifies its additional operational overhead.
What is a RAG pipeline?
A retrieval-augmented generation pipeline transforms unstructured data-such as PDFs, emails, contracts, and internal documents-into a format that an AI model can search and use to generate answers. The system extracts text from the source documents, divides it into chunks, and generates embeddings. These results are then stored in a vector database. When a user poses a question, the pipeline retrieves the most relevant chunks to supply context for the response. This process is suitable for internal question-and-answer tools and compliance applications where inaccurate information could cause significant consequences.
What is Lambda architecture in data pipeline design?
Lambda architecture runs a batch layer and a streaming layer at the same time. It merges the results from both layers in a serving layer to deliver historical accuracy together with real-time data. The batch layer processes all historical data at regular intervals to generate complete views. The speed layer supplies real-time estimates to fill the gap between batch updates, and these estimates are replaced when the next batch run completes. This approach demands maintaining two separate codebases to perform the same calculations, increasing engineering effort.
What is Kappa architecture?
Kappa architecture eliminates the batch layer present in Lambda architecture and processes all data via a single streaming pipeline. To correct historical results, users replay events from a durable log through the same pipeline rather than using a separate batch system. This approach suits teams with established streaming infrastructure that need real-time analytics while avoiding the maintenance of two codebases. It works well when historical reprocessing occurs infrequently enough to be handled via log replay.