A pipeline is the least glamorous part of any data stack and the part that quietly decides whether the rest of it works. Get the plumbing right and your models, dashboards and AI features run on data you can trust. Get it wrong and you spend your weeks chasing why a number moved instead of building anything.
We build data and AI systems at Shipshape Data, which means we spend a lot of time inside these tools, and a fair amount of it cleaning up after ones picked for the wrong reasons. A tool that shines in a five-minute demo can still buckle the first time it meets real volume, a flaky source API, or a team without the people to run it. So this is not a ranking. Ranking twelve tools one to twelve would be dishonest, because they do different jobs and the right pick depends far more on your stack than on any league table.
Instead we have grouped these twelve by the job they do: orchestration, ingestion, transformation and streaming. For each one you get what it is, where it sits, who it suits, and the catch nobody mentions in the marketing. Read the two or three that match your situation and skip the rest.
The four jobs a pipeline actually does
One thing worth getting straight before the list, because it saves you from buying the wrong category of tool. A pipeline breaks into four layers, and most of these tools do one of them well rather than all four adequately.
- Ingestion: getting raw data out of sources (APIs, databases, SaaS apps) and into your warehouse or lake. Airbyte, Meltano, Singer, Debezium.
- Transformation: shaping that raw data into something usable once it has landed. dbt Core.
- Orchestration: deciding what runs, in what order, and what happens when a step fails. Airflow, Dagster, Prefect, Luigi, Kedro.
- Streaming: moving data continuously rather than in scheduled batches. Kafka, and NiFi for flow-based routing.
Almost every stack we build combines tools from three or four of these, not one product that claims to do the lot. When a tool sounds appealing, the first question is not "is it good" but "does it do the job I have".
Orchestration: what runs, in what order
Orchestrators do not move data. They conduct: they decide the sequence, handle retries when something falls over at 3am, and give you somewhere to look when it does. More than a handful of dependent steps and you want one.
1. Apache Airflow
Airflow is the one everyone has heard of, and for good reason. It came out of Airbnb, went to the Apache Software Foundation, and is now the default orchestrator in a huge number of production stacks. You define pipelines as code, in Python, writing Directed Acyclic Graphs (DAGs) that lay out which task depends on which, and Airflow handles the scheduling, retries and dependency resolution from there.
The strength is reach. Its provider ecosystem covers hundreds of integrations, from AWS and Google Cloud to Snowflake and dbt, so it plugs into almost anything. The UI is mature and tells you what you need: pipeline health, task durations, where it failed.
Airflow's code-first design is a real advantage for engineering teams, and a real tax on everyone else. You need solid Python to get value from it.
What the demos skip is the running cost. In production Airflow means a scheduler, workers, a metadata database, and often a message broker on top. For a team with infrastructure people that is fine; for a smaller team without dedicated platform engineering, it becomes a second job. We tend to steer those teams toward Cloud Composer on Google Cloud or Amazon MWAA, both of which host Airflow so you carry less of that weight.
2. Dagster
Dagster is younger than Airflow and has earned a following among teams who care about code quality. It flips the mental model. Instead of thinking in tasks, you think in assets: the tables, files and models your pipeline produces. Dagster understands the relationships between those outputs, not just the order of the steps that make them, which is why it comes with proper data lineage almost for free.
That asset-centric model makes freshness and dependencies much easier to reason about across a tangled pipeline. Built-in testing, type-checking and partitioned runs suit teams who treat pipeline code as seriously as application code.
If your team already follows software engineering discipline, Dagster feels like it was built for you. If it does not, the learning curve will bite.
The catch is that curve. The concepts take upfront investment to click, and the community and connector ecosystem, though growing, is not yet as broad as Airflow's. Install it with a single pip command, get the Dagit UI straight away, then move to Kubernetes or Docker for production. Free under Apache 2.0, with a managed cloud version if you want the control plane handled.
3. Prefect
Prefect pitches itself as the friendlier Airflow, and it largely delivers. You wrap ordinary Python functions in decorators and they become tracked, observable tasks, with very little boilerplate in between. Migrating a pile of existing scripts into a real orchestrated pipeline is genuinely quicker here than almost anywhere else.
Developer experience is the standout. Dynamic workflows that change at runtime are straightforward, and the built-in observability shows task states, logs and failures without bolting on extra tooling. An engineer new to orchestration can have something working the same afternoon.
Its hybrid model also keeps your data processing inside your own infrastructure while Prefect Cloud handles coordination, which matters the moment data residency is on the table.
Deeper customisation and hardened production setups still ask for solid Python and infrastructure knowledge, and some advanced scheduling and concurrency patterns carry their own curve. It suits teams running batch processing, API ingestion and ML experiment workflows who want orchestration without Airflow's full operational weight. Prefect Core is free under Apache 2.0; Prefect Cloud adds a managed control plane with a free tier.
4. Luigi
Luigi is the old hand. Spotify built it to run long batch jobs with gnarly dependency chains, and although the newer orchestrators have taken most of the attention, Luigi still runs quietly in production at plenty of places because it is simple and it does not ask for much.
Simplicity is the whole pitch. No distributed architecture to babysit, a dependency model you can hold in your head, and a central daemon that tracks task state. For a team that wants basic orchestration without Airflow's or Dagster's infrastructure, it covers the ground without fuss.
It also shows its age. There is no scheduler of its own, observability is thin, dynamic task generation is not really there, and the development community has gone quiet. If your pipelines are scheduled batch jobs with static dependencies and no branching, that minimal footprint is an advantage; push past that and you hit the ceiling fast. It suits straightforward batch ETL and teams sitting on a working Luigi codebase that does not yet justify a migration. Free under Apache 2.0, no commercial tier.
5. Kedro
Kedro is the odd one here, less an orchestrator than a way to structure machine learning projects so they survive contact with production. QuantumBlack (now part of McKinsey) built it, the Linux Foundation maintains it, and its reason to exist is turning notebook-and-script ML work into maintainable Python projects.
It gives you a consistent project layout, separates configuration from code, and makes ML pipelines testable and reproducible in a way notebooks almost never are. The value shows up most at handoff, when a data science team passes pipeline code to data engineering without a full rewrite. We have seen that handoff eat weeks of rework on teams that skipped this kind of structure.
Kedro is narrow on purpose. It structures ML pipelines; it does not replace a scheduler or an ingestion platform, so you wrap Kedro pipelines inside Airflow or Prefect for scheduling. Install with pip, scaffold a project from the CLI, done. Free under Apache 2.0, no commercial tier.
Ingestion: getting raw data in
Ingestion is the least appreciated and most fiddly layer. Every source has its own quirks, rate limits and ways of breaking. The tools here exist so you are not hand-writing and forever maintaining extraction scripts for forty different APIs.
6. Airbyte
Airbyte handles the EL in ELT: extract and load, not transform. It gives you a large library of pre-built connectors that handle the tedium of pulling data out of APIs, databases and SaaS tools and dropping it into your warehouse or lake. You sit it upstream of your transformation tool, usually dbt Core, so raw data lands before your logic runs, with an orchestrator like Airflow or Dagster on top.
The connector catalogue is one of the largest going, from Postgres and Salesforce to custom REST APIs you define yourself, and the connector development kit lets your team build and maintain bespoke connectors without forking the whole project.
The trade is real: running Airbyte in production wants meaningful infrastructure, and connector quality varies across the catalogue. Test the connectors for your specific sources before you commit to them.
It fits when you need to consolidate many disparate sources into one warehouse, and it suits analytics platforms or AI data foundations that depend on reliable, repeatable loads. Deploy with Docker Compose locally or Kubernetes in production; Airbyte Cloud removes the self-hosting burden if you prefer. Core is free under the MIT licence; Cloud is usage-based.
7. Meltano
Meltano, originally out of GitLab, is CLI-first ELT that wraps the Singer specification (next on the list) with project management, version control and environment config, turning a loose standard into a structured, reproducible project.
It handles the EL through Singer-compatible connectors, called taps and targets, and hands transformation to dbt Core. The advantage is that your whole pipeline setup lives in a Git repository, so you get reproducibility and auditability that ad hoc ingestion scripts rarely manage, and adding a source is often one line.
The catch sits underneath it: connector quality depends entirely on the Singer ecosystem, which is uneven. Some taps are well maintained, others lag behind the API changes they track. Meltano suits analytics engineering teams who want a Git-native approach to ELT. It works well for:
- Version-controlled ingestion pipelines pulling SaaS data into a central warehouse
- Running dbt transformations after loads inside one coordinated project
Install with pip, initialise a project in minutes, and let Airflow or Dagster handle scheduling on top. Free under the MIT licence, no commercial tier.
8. Singer
Singer is not a platform at all. It is a specification, from Stitch Data, that defines how extraction scripts (taps) and loading scripts (targets) talk to each other: simple JSON over Unix pipes, so any compliant tap can pair with any compliant target. Tools like Meltano are built directly on top of it.
The strength is reuse. Because any tap works with any target, the community has contributed hundreds of open-source connectors, and if all you need is a lightweight extraction script, Singer taps run happily on their own.
The limit follows from what Singer is. It is a specification, not a system, so there is no scheduling, no monitoring and no error handling beyond what you build yourself.
It fits lightweight, scriptable extraction where you do not want a heavier platform, and teams building custom connectors to reuse across frameworks. You install taps and targets as Python packages and wire them with pipes; most orchestrators that support Singer handle that wiring in production. Free under Apache 2.0.
9. Debezium
Debezium does change data capture. It watches your operational databases and streams every row-level change out as an event, built on top of Apache Kafka, without touching your application code. That makes it a proper fit when you need to react to changes as they happen rather than polling a table on a timer.
It sits between your operational databases and whatever comes downstream, reading straight from the transaction logs (the PostgreSQL WAL, the MySQL binlog) and emitting change events to Kafka topics. Most teams run it alongside Kafka Connect and Kafka Streams for the full CDC pipeline. The strength is low-latency, accurate capture that barely touches your source database, because reading the log is far lighter than repeatedly querying the tables.
In practice it is one of the more dependable ways to keep a warehouse or AI system in step with operational data, without hammering production with full-table extracts on a schedule.
The obvious cost is that it needs Kafka underneath, a lot of infrastructure to stand up if you are not already running a cluster. It suits continuous sync between operational databases and analytics or AI systems, and feeding live data into machine learning pipelines that need current inputs. You deploy it as Kafka Connect connectors, one per database type (PostgreSQL, MySQL, MongoDB, SQL Server), each with its own log config to sort first. Free under Apache 2.0.
Transformation: shaping data once it lands
10. dbt Core
dbt Core is the one tool on this list that does transformation and nothing else, and that focus is exactly why it took over. It lets your analytics and engineering teams transform raw data inside the warehouse using plain SQL SELECT statements. It does not move data; it defines, tests and documents your transformation logic as version-controlled code, so your models get the same discipline you would give application software.
It sits at the transformation layer, after raw data has landed, and works alongside ingestion tools like Airbyte or Meltano to form the T in ELT. Most teams trigger dbt runs through an orchestrator rather than alone. What you get is version control, automated testing and auto-generated documentation that keeps everyone honest about what each model produces and why.
It became the default in analytics engineering for a simple reason: it makes big, messy SQL codebases manageable at scale.
Its limit is its scope. dbt handles transformation only, so you still need separate ingestion and orchestration for a complete pipeline. It suits analytics engineering teams building transformation layers on BigQuery, Snowflake or Redshift, anywhere modelling, testing and documentation need to scale with data volume. Install it with pip plus your warehouse adapter, and run it as an orchestrated job. Free under Apache 2.0; dbt Cloud adds a hosted IDE, scheduler and observability on paid plans.
Streaming: moving data continuously
Batch is fine for most things. But some problems need data in motion, not data that arrives every hour on the hour: fraud detection, live personalisation, keeping a warehouse in step with production in near real time. That is where the last two come in.
11. Apache Kafka
Kafka is the backbone of most real-time architectures. LinkedIn built it, Apache took it on, and it does one thing at enormous scale: high-throughput, fault-tolerant movement of data streams. Where the batch tools above run on a schedule, Kafka runs continuously.
It acts as a durable, ordered log of events that producers write to and consumers read from, decoupling your sources from processing and storage so an upstream change does not knock everything downstream over at once. Teams usually run it alongside Debezium for change data capture or Apache Flink for stream processing. The strength is raw capability: massive event volumes at low latency, with a replication model that survives node failures.
Kafka's power comes with weight. A production cluster needs experienced hands to tune brokers, watch consumer lag and handle partition rebalancing without drama.
It suits architectures that need continuous, high-volume movement rather than batch loads:
- Real-time event streaming and activity tracking at scale
- Change data capture pipelines feeding warehouses from operational databases
- Live ingestion into ML systems that need current inputs
Run it locally with Docker Compose for development; in production, Kubernetes with the Strimzi operator handles cluster management far more cleanly than a manual setup. Free under Apache 2.0. Confluent offers a managed cloud version with a free tier if you would rather not run the cluster yourself.
12. Apache NiFi
NiFi is the visual outlier here. The NSA built it, Apache open-sourced it, and it is built around drawing data flows in a browser rather than writing them as code. It handles near-real-time movement between sources and destinations: you pull from APIs, databases, file systems or streams and route the data through transformation and filtering processors before it goes downstream. It does not sequence tasks the way Airflow or Dagster do; it manages a continuous flow of records between connected components.
The visual, flow-based interface makes it reachable for operators and engineers who would rather not write much code. It handles back-pressure, provenance and guaranteed delivery natively, which earns its keep where a lost or duplicated record causes downstream pain.
Its built-in provenance gives you a full audit trail of where each record came from and what happened to it, which is hard to reproduce in code-first tools without extra instrumentation.
The cost is weight and maintainability. NiFi wants dedicated infrastructure, and the visual canvas gets awkward as flows grow, which makes standard version control and testing harder than a code-first team is used to. It works best for real-time ingestion, IoT data collection, and compliance-sensitive environments where auditability and provenance are hard requirements. Run it from the official Docker image or on-premises; clustered production needs ZooKeeper for state coordination. Free under Apache 2.0.
Picking the right stack, not the longest one
No single tool covers every layer, and that is the point. Each solves one job well. The work is choosing the combination that fits your architecture, your team's skills and your data volumes. A common, sensible starting point for analytics teams is Airbyte for ingestion, dbt Core for transformation and Airflow for orchestration, but your constraints should shape that more than any generic recommendation.
Before you commit, map the requirements plainly. What sources do you need to connect. Are your workloads batch or streaming. And the one people skip: how much infrastructure can your team realistically operate. We have inherited enough failed stacks to say it plainly: the impressive-looking one nobody on the team can keep alive is worse than the boring one that runs.
If you want a second opinion before you build, or you have inherited a pipeline that keeps breaking and you are not sure why, that is the work we do. Talk to us and start with a clear picture of your foundation rather than a vendor demo.