The overnight batch job finishes at 4am. By 9am the warehouse team is picking orders against stock numbers that are already twelve hours out of date, and the last three units of a bestselling line get promised to two different customers before anyone notices the mismatch. Nobody built the system badly. It was built to run once a night, and once a night is exactly what it does.
We build data and AI systems for clients at Shipshape Data, and this particular argument comes up on nearly every project, usually early, usually before anyone has agreed what the system is actually for. The mistake we see most is a team reaching for a real-time architecture because it sounds like the modern choice, then six months later maintaining a streaming pipeline for a report that gets read once a week. The opposite mistake happens too, less often but more expensively: a fraud or safety process left on a nightly batch because nobody wanted to touch the pipeline, quietly costing money every single day it stays that way.
This guide covers what batch and real-time processing actually mean, where the cost and complexity of streaming shows up once you commit to it, the micro-batching compromise most teams end up reaching for anyway, what a genuine streaming architecture involves running day to day, and why the sensible default for most organisations is batch, with real-time earned rather than assumed.
Two ways of moving the same data
Batch processing collects data over a period, a day, an hour, sometimes a week, and processes it all together in one run. The overnight job that refreshes yesterday's sales into this morning's dashboard is batch. So is the payroll run, the monthly reconciliation, most data warehouse loads. Real-time processing, sometimes called stream processing, handles each event as it arrives, individually or in tiny slices, so the system reacts within seconds rather than waiting for the next scheduled run.
What batch actually buys you
Batch is simple to reason about. A job runs, it either succeeds or fails, and if it fails you rerun it. You can look at the whole dataset at once, which makes some calculations genuinely easier: a monthly total, a full deduplication pass, a join across several large tables. Batch jobs are also cheap relative to what they do, because the compute only spins up for the duration of the run rather than sitting there listening for events around the clock.
The trade is staleness. Whatever the batch produces is only as current as the last run, and the gap between runs is a gap the business has to live with. For most reporting, that gap is fine. Nobody needs last month's revenue figure to the second.
What real time actually buys you
Real-time processing keeps the gap close to zero. An event happens, the system reacts, and the reaction is available before the next thing happens. This matters when a decision has to be made against the current state of the world rather than a snapshot from an hour ago: blocking a fraudulent card transaction before it clears, adjusting a price against live demand, alerting an engineer the moment a machine starts drawing more current than it should.
Most things sold as "real time" are actually near real-time, seconds rather than milliseconds, and that distinction is worth holding onto. True sub-second processing is a specific engineering commitment. Most business problems do not need it, they need something faster than overnight, which is a much easier and cheaper bar to clear.
The cost of being fast when you don't need to be
Streaming infrastructure is not simply the batch pipeline with a shorter interval. It is a different way of building software, and the difference shows up in cost, in headcount, and in how many things can quietly go wrong at three in the morning.
The infrastructure tax
A batch job can run on infrastructure that exists only for the duration of the run. A streaming pipeline has to exist permanently: a message broker, consumers that are always listening, state that has to be held somewhere while events are being processed rather than sitting comfortably in a finished table. That infrastructure needs monitoring around the clock, because a batch job that fails at 2am gets noticed at 9am when someone checks the dashboard, while a streaming pipeline that fails at 2am is dropping live events for seven hours before anyone looks.
There is also a people cost that rarely makes it into the original business case. Somebody has to be on call for the pipeline itself, in addition to the application sitting on top of it. Somebody has to own the broker, patch it, plan its capacity, and understand what happens when a consumer group falls behind during a traffic spike. A batch job that breaks can usually wait until morning. A streaming pipeline that breaks is, by definition, breaking right now, and the team has to be built around that fact rather than discovering it after go-live.
The failure modes multiply
Batch failure is usually clean. The job did not complete, you fix the cause, you rerun it, and the world is exactly as it would have been if nothing had gone wrong. Streaming failure is messier. Events arrive out of order. A consumer falls behind and has to catch up while new events keep arriving. Two events that should have been processed together land in different windows because one was delayed by a few hundred milliseconds on the network. None of these are exotic edge cases. They are the normal operating conditions of a distributed system, and a team that has only ever run batch jobs tends to be caught out by how much of streaming engineering is really about handling disorder gracefully.
Real-time architecture does not remove complexity, it moves it from "when the batch job runs" to "at every single moment the system is switched on", and somebody has to be awake for that.
None of this is an argument against streaming. It is an argument for knowing what you are signing up for before you build it, because the honest cost of a real-time system is not the initial build. It is the years of on-call after.
When batch quietly costs you more
Batch is the sensible default, but default is not the same as always correct, and there are cases where a nightly refresh is simply the wrong answer, whatever it costs to admit that after the fact.
The tell is the decision, not the data
The question worth asking is not how fresh the data could theoretically be. It is what decision depends on that freshness, and what it costs when the decision is made on stale information. Fraud is the clean example: a batch job that flags suspicious transactions the next morning has flagged them after the money has already gone. The fraud team still finds the pattern. They just find it too late to stop the specific transaction that mattered. Inventory during a flash sale is another: stock counts that update once an hour are fine on a normal Tuesday and a liability the moment demand spikes past what the last batch run recorded. Predictive maintenance sits in the same territory. A vibration reading that gets analysed in tomorrow's batch is analysing a bearing that may already have failed overnight, and the entire value of the sensor was in catching the change while there was still time to schedule a repair instead of a replacement.
What genuinely changes when you move faster
In each of these cases, the value is not in the data being more accurate. Batch and streaming pipelines can produce identically accurate numbers. The value is entirely in the timing of the action that follows. If nobody acts differently within the gap that streaming closes, moving to real time buys nothing but cost. If somebody does, and that action is worth more than the gap costs to close, that is the case for streaming, and it is worth writing down explicitly rather than assuming.
We ask clients to write the sentence out in full before committing budget: "if we knew this within X seconds instead of within a day, we would do Y, and Y is worth roughly Z." If nobody can finish that sentence, the case for real time has not actually been made yet, however urgent the request sounded in the meeting where it came up.
Micro-batching: the compromise almost everyone reaches for
Between "once a night" and "the instant it happens" sits a wide middle ground that most organisations actually live in without necessarily naming it: micro-batching. Instead of collecting a full day's data before processing, the pipeline runs every few minutes, sometimes every few seconds, processing whatever has accumulated in that short window.
What it actually buys you
Micro-batching gets you most of the practical benefit of real time without the full operational weight of a genuine streaming system. A dashboard that refreshes every five minutes feels current to almost anyone looking at it, and the engineering behind it is much closer to batch than to stream: scheduled runs, bounded windows, failures that are contained to one short interval rather than an open-ended stream of events. This is the level a great many "real-time analytics" dashboards actually operate at, and it is usually the right level, because the business rarely needed true sub-second reaction, it needed the dashboard to stop looking stale.
Where it still falls short
Micro-batching does not help when the requirement is a reaction to a single event as it happens, a fraud check that has to fire before a transaction clears, an alert that has to fire the moment a threshold is crossed rather than up to five minutes later. If the gap between the event and the reaction has a genuine cost attached to it, shrinking the batch interval only shrinks the cost, it does not remove it. At that point you are building a streaming system whether you call it one or not, and it is worth calling it one, because that is the point where the operational commitments described above start applying in full.
What a streaming architecture actually involves
When the case for real time is genuine, the shape of the system changes. It stops being jobs that run and becomes a continuous flow that has to be designed, not scheduled.
The core pattern
Events are published onto a log or broker as they happen: an order placed, a sensor reading, a page view. Consumers subscribe to that log and process events as they arrive, often transforming or enriching them before writing the result somewhere downstream, a database, a dashboard, another system entirely. The log itself usually retains events for a window, which matters more than it sounds, because it means a consumer that falls behind or restarts can replay what it missed rather than losing it. Good streaming design treats "a consumer will eventually fail and need to catch up" as the normal case, not the exception.
The operational reality
Running this well needs schema discipline, because a producer that quietly changes the shape of an event breaks every consumer downstream at once, often without an obvious error message. It needs monitoring that watches lag, the gap between an event arriving and being processed, because lag creeping upward silently is how a streaming system fails slowly rather than loudly. And it needs people who understand event-time versus processing-time, the difference between when something actually happened and when the system got round to handling it, because conflating the two is a reliable source of subtly wrong numbers that take a long time to trace back to the cause. None of this is impossible. It is a genuine specialism, and treating it as a slightly faster version of a batch pipeline is how projects get into trouble.
Most teams do not build this from scratch. Managed message brokers and stream-processing services exist precisely because running a broker reliably is its own job, separate from whatever the business logic sitting on top of it is trying to do. That does not remove the operational reality above, it just moves some of it onto a vendor's shoulders. Somebody still has to design the topics, watch the lag, and decide what happens when a consumer needs to replay three hours of history after a bug fix. Buying the platform buys you out of running the broker. It does not buy you out of understanding the stream.
Why most organisations need both
The realistic end state for almost every organisation we work with is not a choice between batch and streaming. It is both, applied to different things, because the data underneath a business genuinely splits into things that need an immediate reaction and things that do not.
Historical reporting, monthly reconciliation, most machine learning training runs, the bulk of a data warehouse: none of this benefits from streaming, and forcing it through a real-time pipeline adds cost and fragility for no gain anyone will notice. A data pipeline built well for batch, with clear schedules, clean failure handling and sensible monitoring, is not a lesser architecture waiting to be upgraded. It is the correct architecture for what it is doing. The ETL or ELT pattern underneath it barely changes whether the trigger is a nightly schedule or a five-minute micro-batch, the difference sits mostly in orchestration rather than in the transformation logic itself.
Alongside that sits a much smaller set of paths that genuinely need to react in the moment: fraud checks, live pricing, safety alerts, anything where a person or a system has to act before the next scheduled run would even have started. Building both well usually means the streaming paths are narrow and deliberately scoped, sitting next to a much larger batch estate rather than replacing it, and the two increasingly share the same storage underneath through patterns covered in our guide to data pipeline design patterns, so a business does not end up maintaining two entirely separate data platforms for the two speeds it needs.
Where to start
Start by listing the decisions your data actually drives, not the systems you already have opinions about. For each one, ask what happens if the answer is a day old, an hour old, five minutes old. Most of the list will tolerate a day. A short list will not, and that short list is your real-time scope, no bigger.
Build the batch estate first if it is not already solid, because a streaming system bolted onto a shaky batch foundation just moves the shakiness somewhere harder to see. Then treat every candidate for real time as a specific, costed decision: what does the business gain in the gap being closed, and what does it cost to keep a streaming pipeline running and monitored for as long as it exists. Most organisations, once they have actually done that exercise, find the honest answer is smaller than the one they started with.
If you are trying to work out which parts of your data genuinely need to move faster and which are better left on a schedule, talk to us. We would rather help you scope the real-time slice honestly than watch a business pay for a streaming platform that mostly refreshes a report nobody reads until Monday morning.