Data ingestion explained: batch, streaming and CDC

A retailer's nightly batch job used to be enough. Then the marketing team asked for stock levels that update within minutes, the finance team wanted a live feed from the new payments provider, and someone in the warehouse still emails a CSV every Friday. Three sources, three rhythms, one pipeline expected to cope with all of them. This is where most data ingestion problems actually start: not with the technology, but with nobody having decided which sources need to be fast, which can wait, and what happens the day one of them goes quiet.

Data ingestion is the process of moving data from its source, whether a database, an API, a file drop or a sensor feed, into a system where it can be stored, transformed and used. It can run in scheduled batches, as a continuous stream, or through change data capture that tracks row-level updates as they happen.

What data ingestion actually does

Every data platform starts with data arriving from somewhere else. Ingestion is the layer that takes data out of the system where it was created, a point-of-sale terminal, a CRM, a sensor, a partner's API, and lands it somewhere your own team controls, typically a landing area in a warehouse or lake. Nothing gets transformed at this stage. The job is to get a faithful copy of the data across the boundary without losing or corrupting it.

That sounds mechanical, and at a small scale it is. The complexity shows up once you have dozens of sources on different schedules and reliability, each a potential point of failure nobody notices until a report looks wrong. Getting ingestion right is less about clever tools and more about being honest about which sources need to be fresh, which can lag a day, and who is accountable when one breaks.

Batch ingestion

Batch ingestion moves data in scheduled chunks, hourly, nightly or weekly, rather than as it happens. A job extracts everything new or changed since the last run, often using a timestamp column or a full reload, and writes it to the target. For most reporting workloads this is still the right default: simple to build, cheap to run, and easy to reason about when something goes wrong, since you can rerun the job and get the same result.

The trade-off is staleness. If a batch runs at 2am, a change made at 9am the previous day is invisible until the next run finishes. That is fine for a monthly finance report and hopeless for a system trying to stop overselling stock. Batch also tends to hit source systems hardest when the job kicks off, which is why nightly loads are often scheduled for quiet hours.

Streaming ingestion

Streaming ingestion moves records as they are created, one event or a small micro-batch at a time, usually through a message broker such as Kafka or a managed equivalent. Instead of waiting for a schedule, downstream systems pick up new data within seconds. This matters when a decision depends on the current state of something: fraud checks, live inventory, dashboards people actually watch during the day rather than review the morning after.

Streaming buys freshness at the cost of complexity. The system is always running, and so always capable of falling behind or dropping messages if a consumer slows down. Ordering, duplicate events and partial failures all need handling a nightly batch job never has to think about. Build it where freshness genuinely changes what someone does with the data, not because streaming sounds more advanced.

Change data capture

Change data capture, CDC, is a specific way of feeding a stream: instead of querying a source table and comparing it to last time, CDC reads the database's transaction log and picks up every insert, update and delete as it is committed. That gives a low-impact, near-real-time feed without hammering the source, and it captures deletes properly, which timestamp-based batch jobs routinely miss.

CDC suits systems of record that change constantly and where downstream tables need to mirror the source closely: an orders table feeding a warehouse, a customer record feeding a data lake. It needs more setup than a batch pull, including log access and a tool such as Debezium, plus a plan for a connector that falls behind. Where a daily view is genuinely enough, CDC is more engineering than the problem calls for.

Where the data actually comes from

Sources rarely arrive in one shape. Operational databases sit behind CDC or scheduled queries. SaaS tools such as a CRM expose a REST API with their own rate limits and pagination quirks. Files still turn up: CSV exports, a supplier's daily feed dropped into an SFTP folder, a spreadsheet someone emails because the proper integration was never built. Event streams come from application logs, website tracking and sensors, often at far higher volume than expected.

Each source type carries its own failure shape. APIs throttle you or change schema without notice. Files arrive late, twice, or with a different delimiter. Sensor feeds go quiet when a device loses connectivity, which looks identical to nothing to report unless you are watching for it. A pipeline built for only one of these patterns tends to work fine in testing and fail quietly in production the first time a source behaves differently.

Data ingestion versus data integration and extraction

The terms get used loosely, and it is worth being precise. Ingestion is the act of getting data from a source into your environment. Data integration is the broader job of combining data from multiple sources so it works together: matching customer records across systems, resolving conflicting field names, agreeing a single version of an entity. Ingestion is one step inside integration, not a stand-in for the whole job.

Extraction is closer still, and the confusion is often about which half of a job a tool is doing. Extraction is the read, pulling data out of the source system. Ingestion covers extraction plus landing that data somewhere usable, so extraction is the first half of ingestion. In an ETL or ELT pipeline, ingestion is the E, and everything that turns raw data into something trustworthy happens afterwards, in transform and load.

Where ingestion pipelines actually fail

The failures that hurt are rarely dramatic outages. A source renames a column and the pipeline quietly drops it instead of erupting into an obvious error. A batch job runs twice after a retry and duplicates every row for that day. A field that used to be an integer starts arriving as text, and everything downstream that assumed a number quietly breaks. None of this shows in a dashboard built only to watch whether the job ran.

The other common pattern is volume and timing. A source that sent a thousand records a day sends a million after a marketing campaign, and a job that finished in ten minutes now overruns and collides with the next run. Or a source goes offline for maintenance and the job either fails loudly, which is manageable, or succeeds with an empty result set that looks like a quiet day. Checks for row counts, schema and freshness matter more than most teams budget for.

Getting ingestion right from the start

The pipelines that hold up long term are designed around the actual shape of each source rather than one pattern applied everywhere. That means batch where daily is enough, CDC or streaming where freshness changes a decision, and monitoring built in from day one rather than bolted on after the first silent failure. It also means someone owns each source, so when it changes shape, and eventually it will, a person finds out before the finance team does.

This part is easy to underestimate from outside: ingestion is rarely the interesting bit of a data project, so it gets the least attention and the most technical debt. Treating it as part of a data foundation built for AI, rather than plumbing to be minimised, is usually the difference between a platform that stays trustworthy as sources multiply and one that needs firefighting every time a new system connects.

Frequently asked questions

Want a straight view of where AI can help your business first? Talk to us and start with a clear picture instead of a vendor demo.