The overnight batch job completes at 4am. By 9am, the warehouse team is fulfilling orders using stock numbers that are twelve hours old. The last three units of a popular product are then committed to two separate customers before the discrepancy is identified. The system was not developed with errors; it is designed to run once per night, and it operates as intended.
Batch and real-time processing are two methods for moving data. Batch processing collects records over a specific period and processes them as a group. Real-time systems process each event as it occurs. Batch processing is appropriate for reporting and data warehousing. Real-time processing is useful when decisions require immediate action, such as fraud detection or live pricing updates.
At Shipshape Data, we build data and AI systems for our clients. This specific argument occurs during almost every project. It usually happens early, before the purpose of the system has been defined. The most common mistake is when a team chooses a real-time architecture because they believe it is the modern standard. Six months later, they are forced to maintain a streaming pipeline for a report that is only read once a week. The opposite mistake is less common but more expensive. This occurs when a fraud or safety process remains on a nightly batch cycle because the team avoids changing the pipeline. This delay costs money every day the system remains unchanged.
This guide explains the definitions of batch and real-time processing. It describes the costs and technical difficulties that occur after adopting streaming, and the micro-batching methods that many teams use instead. It also outlines the daily operational requirements of a true streaming architecture. The text argues that most organisations should use batch processing by default and only move to real-time systems when specific needs justify the transition.
Two methods for transferring the same data
Batch processing collects data over a period of time-a day, an hour, or sometimes a week-and then processes it all together. For instance, a job that updates sales data overnight for a morning dashboard uses batch processing. Payroll processing, monthly reconciliations, and most data warehouse loads are also completed in batches. Real-time processing, also called stream processing, handles each event as it occurs, either one at a time or in small groups, so the system can respond within seconds. It doesn't need to wait for a set processing time.
What batch buys you
Batch is simple to reason about. A job runs, it either succeeds or fails, and if it fails you rerun it. You can look at the whole dataset at once, which makes some calculations genuinely easier: a monthly total, a full deduplication pass, a join across several large tables. Batch jobs are also cheap relative to what they do, because the compute only spins up for the duration of the run. It isn't sitting there listening for events around the clock.
Batch processing results in data that is only as current as the most recent run. The business must operate with the delay that occurs between these runs. This delay is acceptable for most reporting. Monthly revenue figures do not need to be accurate to the second.
The benefits of real-time systems
Real-time processing maintains a minimal delay between an event and a response. The system reacts to an occurrence and provides the result before the next event begins. This capability is necessary when decisions must rely on current data instead of older records. Examples include stopping a fraudulent credit card transaction before it is completed, changing a price based on immediate demand, or notifying an engineer as soon as a machine exceeds its normal electrical current.
Most products marketed as real-time are actually near real-time. These systems operate in seconds rather than milliseconds. This distinction is important to maintain.
Processing data in less than one second requires a specific engineering approach. Most business requirements do not demand this level of speed. Most organisations only need systems that operate faster than overnight processing, which is a less expensive and less difficult goal to achieve.
The cost of being fast when you don't need to be
Building software with streaming infrastructure requires a different approach than using batch pipelines that run on short intervals. These differences affect expenses, staffing requirements, and the frequency of technical failures that occur outside of business hours.
The infrastructure tax
A batch job can run on infrastructure that exists only for the duration of the run. A streaming pipeline has to exist permanently: a message broker, consumers that are always listening, state that has to be held somewhere while events are being processed, not sitting comfortably in a finished table. That infrastructure needs monitoring around the clock, because a batch job that fails at 2am gets noticed at 9am when someone checks the dashboard, while a streaming pipeline that fails at 2am is dropping live events for seven hours before anyone looks.
Business cases also rarely include the cost of the impact on people.
Someone must be responsible for the pipeline itself, as well as the application it supports. This person needs to manage the broker, apply patches, plan for capacity, and understand the consequences when a consumer group lags during high traffic. While a failed batch job can typically be addressed the next morning, a streaming pipeline failure is an immediate issue. The team must be structured to handle these real-time requirements before the system goes live.
The failure modes multiply
Batch failures are typically straightforward. You can fix the error and rerun the process if a job does not complete. The final state is the same as if the error had never occurred. Streaming failures are more complex. Events often arrive out of order. A consumer might fall behind and need to process a backlog while new events continue to arrive. Two events that belong together might end up in different processing windows because of a short network delay. These situations are standard conditions for distributed systems. Teams accustomed to batch processing are often surprised by how much streaming engineering focuses on managing these types of disorder.
Real-time architecture does not eliminate complexity. Instead, it shifts the operational burden from "when the batch job runs" to "at every single moment the system is switched on". This change requires personnel to be available for monitoring and maintenance at all times.
These points do not constitute an argument against streaming.
You should understand the requirements before you begin construction. The actual cost of a real-time system includes the years of on-call duties that follow the initial development phase.
When batch processing increases your costs
Batch processing is usually the standard choice. However, a standard choice is not always the correct one. In some cases, a nightly refresh is the wrong solution, regardless of the difficulty of acknowledging that error later.
The tell is the decision, not the data
The question worth asking is what decision depends on that freshness, and what it costs when the decision is made on stale information. How fresh the data could theoretically be is beside the point. Fraud is the clean example: a batch job that flags suspicious transactions the next morning has flagged them after the money has already gone. The fraud team still finds the pattern. They just find it too late to stop the specific transaction that mattered. Inventory during a flash sale is another: stock counts that update once an hour are fine on a normal Tuesday and a liability the moment demand spikes past what the last batch run recorded. Predictive maintenance sits in the same territory. A vibration reading that gets analysed in tomorrow's batch is analysing a bearing that may already have failed overnight, and the entire value of the sensor was in catching the change while there was still time to schedule a repair instead of a replacement.
The actual effects of increasing your speed
In these cases, the value depends on the timing of the subsequent action rather than on data accuracy. Batch and streaming pipelines can yield the same results. If no one alters their behaviour during the time saved by streaming, switching to real-time processing only raises costs. If someone does act, and that action is worth more than the cost of closing the time gap, then streaming is justified. Document this reasoning explicitly rather than assuming it.
We require clients to fill in a specific sentence before we approve a budget: "if we knew this within X seconds, not within a day, we would do Y, and Y is worth roughly Z." If no one can complete that sentence, the case for real-time processing is not yet established. This applies even if the request appeared urgent in the initial meeting.
Micro-batching: the compromise most people choose
Many organisations work in a middle ground between "once a night" and "the instant it happens" without assigning a specific term to it. This practice is known as micro-batching. The pipeline does not wait for a full day of data to accumulate before starting processing. Instead, it runs every few minutes or every few seconds to handle the data gathered during that short interval.
What it buys you
Micro-batching offers the functional benefits of real-time processing while avoiding the operational complexity of a streaming system. A dashboard that refreshes every five minutes looks current to most users. The engineering needed for this method resembles batch processing since it employs scheduled runs, bounded windows, and failures that remain within a single interval instead of impacting a continuous stream. Many "real-time analytics" dashboards operate in this manner. This is usually the right approach because most businesses do not need sub-second reaction times. They only need the data to stay fresh.
Remaining deficiencies
Micro-batching is ineffective when you need to react to a single event immediately. This includes fraud checks that must finish before a transaction clears or alerts that must fire the moment a threshold is crossed instead of minutes later. If a delay between an event and a reaction incurs a specific cost, shortening the batch interval merely reduces that cost rather than removing it. In such cases, you are constructing a streaming system no matter what terminology you apply. You should recognise it as a streaming system because this is when you must fulfil all the operational commitments previously described.
The components of a streaming architecture
When a system needs real-time processing, its architecture changes. It no longer operates as a series of scheduled jobs. Instead, it turns into a continuous flow that needs specific design rather than simple scheduling.
The core pattern
Systems publish events to a log or broker at the moment they occur. Examples are orders, sensor readings, and page views. Consumers subscribe to these logs in order to process events. They frequently transform or augment these events before forwarding them to a database, a dashboard, or another system. The log usually retains events for a defined period. This retention enables a consumer that stops or lags behind to replay the events it missed. Effective streaming design assumes that "a consumer will eventually fail and need to catch up" is a normal part of the process rather than an unusual error.
The operational reality
Operating this system well requires strict schema management. If a producer changes an event's structure, every downstream consumer will fail immediately, often without giving a clear error message. The system also needs monitoring that tracks lag, which is the time gap between an event's arrival and its processing. Rising lag indicates that a streaming system is degrading gradually rather than crashing. Additionally, staff must grasp the distinction between event time and processing time. Event time is when an occurrence actually happened, whereas processing time is when the system handled it. Mixing these two concepts produces inaccurate data that is hard to troubleshoot. These requirements are manageable, yet they represent a specific area of expertise. Projects often fail when teams treat streaming systems as slightly faster versions of batch pipelines.
Most teams do not create these systems from scratch. Managed message brokers and stream-processing services exist because running a broker reliably is a separate task from writing business logic. Using these services does not remove operational requirements. It shifts some responsibilities to a vendor. You still need to design the topics, monitor lag, and decide how to replay history after fixing a bug. Purchasing a platform removes the need to run the broker, but you must still understand how the stream works.
Why most organisations require both
Most of the organisations we work with eventually adopt both batch and streaming processing. They use these methods for different tasks since business data naturally falls into two categories: information that needs an immediate reaction and information that does not.
Historical reporting, monthly reconciliation, most machine learning training runs, and the majority of a data warehouse gain no advantage from streaming. Pushing these processes into a real-time pipeline raises costs and fragility while delivering no clear benefits. A data pipeline built for batch processing, equipped with clear schedules, reliable failure handling, and appropriate monitoring, is the proper architecture for these tasks. It is not a subpar system that needs upgrading. The underlying ETL or ELT pattern stays largely unchanged whether it is launched by a nightly schedule or a five-minute micro-batch. The main distinction lies in the orchestration rather than the transformation logic.
A smaller set of processes also requires immediate reactions. These comprise fraud checks, live pricing, and safety alerts, and a person or system must act before the next scheduled batch run begins. When constructing both systems efficiently, streaming paths are typically narrow and specific. They sit alongside a larger batch infrastructure instead of replacing it. Both systems often share the same underlying storage by applying the methods described in our guide to data pipeline design patterns. This method prevents a business from needing to maintain two separate data platforms for its different speed requirements.
Where to start
Start by listing the decisions that your data informs. Do not begin with the systems you currently use or prefer. For each decision, assess the consequences if the data is one day old, one hour old, or five minutes old.
Most items on your list can be delayed for a day. A small number of specific tasks cannot be delayed. These tasks constitute your real-time scope, and you should not add anything else to it.
Set up a reliable batch system before adding streaming. If the underlying batch infrastructure is unstable, adding a streaming layer will only make those problems harder to identify. Assess each potential real-time project as an individual financial decision. Identify the specific business value obtained by reducing latency and compare it to the long-term costs of maintaining and monitoring a streaming pipeline.
After completing that exercise, most organisations discover that the actual number is smaller than their initial estimate.
If you are trying to determine which parts of your data need faster processing and which can stay on a schedule, talk to us. We prefer to help you accurately define the scope of real-time requirements rather than have a business invest in a streaming platform that mainly updates a report that few people review until Monday morning.
Theory is the first step. If you want to apply this to your own data, our data consultancy will help you move from the article to production.
Frequently asked questions
What is the difference between batch processing and real-time processing?
Batch processing gathers data over a specific period, such as an hour, a day, or a week, and processes it in a single run. An example is an overnight job that updates a dashboard. Real-time processing handles each event as it occurs and reacts within seconds. It does not wait for a scheduled run. Batch processing is simpler and less costly to operate. Real-time processing reduces the time between an event occurring and the system responding to it.
How does real-time processing compare to batch and stream processing?
Real-time processing and stream processing are effectively the same: events are handled individually or in tiny slices as they arrive. They do not wait for a full batch to accumulate. Streaming requires infrastructure that runs permanently, a broker, listening consumers, and monitoring around the clock, which is a different operational commitment than a batch job that only runs during its scheduled window.
Is real-time data always better than batch processing?
Real-time data is only worth the expense if a decision depends on the speed it provides. Examples include stopping a fraudulent transaction or responding to a sudden stock increase during a sale. If no one changes their actions based on that immediate information, switching to real-time systems raises costs without providing a practical advantage.
What does a genuine streaming architecture involve?
A streaming architecture publishes events to a log or broker at the time they occur. Consumers subscribe to these events and process them upon arrival. This system requires schema management, lag monitoring, and a distinction between event time and processing time. Because consumers will eventually fail and need to recover, the design must treat catching up as a standard operation rather than an error.
Why do most organisations need both batch and real-time processing?
Most business data functions well with batch processing. This includes historical reporting, monthly reconciliations, and most warehouse loads, as these tasks do not require immediate updates for decision-making. A smaller number of processes must respond instantly, such as fraud detection or live pricing. The practical approach is to maintain a small, specific real-time layer alongside a larger batch system rather than trying to replace the batch infrastructure entirely.