Data lakehouse architecture: definition, layers, and diagrams

If you ask five employees at your company to define a data lakehouse, you will probably get five different, partially correct answers. That lack of clarity leads to unnecessary costs.

Teams keep building both a data lake and a data warehouse. This creates double the infrastructure costs for a setup that was meant to be a single system.

Data lakehouse architecture merges the flexible, low-cost storage of a data lake with the reliability and query performance of a data warehouse, employing open table formats such as Delta Lake or Apache Iceberg to provide transactions and schema enforcement. Five layers, ingestion, storage, metadata, API, and consumption, work together so a single copy of data serves both BI and AI teams.

Shipshape Data constructs data foundations. Most of our clients currently enquire about lakehouses. Many have read that lakehouses resolve the conflict between data lakes and data warehouses, yet they often lack a clear understanding of the technology. This guide explains what a lakehouse is and why it is useful. It describes the five layers of the architecture, how to plan a project, and common reasons for failure. You can read the whole guide if you are new to the topic, or you can go straight to the section on layers for technical details.

The definition of a data lakehouse

A data lakehouse integrates the low cost and flexibility of a data lake with the performance and reliability of a data warehouse. You can store structured tables, semi-structured logs, and unstructured documents in open formats on inexpensive cloud storage while maintaining fast SQL access, data quality enforcement, and transaction support. This summarises the concept. Most misunderstandings about lakehouses occur when people overlook one of these two components.

Standard data lakes provide inexpensive and flexible storage, yet they lack reliability. Any user can write data, schemas often change without control, and simultaneous write operations can corrupt tables. These issues often cause a loss of data trust within six months. Data warehouses provide reliability but charge high storage rates and use proprietary formats. This vendor lock-in makes it difficult to run machine learning models. A lakehouse tries to combine the benefits of both systems while avoiding their disadvantages. Modern table formats such as Delta Lake, Apache Iceberg, and Apache Hudi enable this architecture. They sit on top of files in object storage to provide the transaction logs, schema enforcement, and versioning capabilities that previously required a warehouse.

This means you no longer copy data between two platforms for different audiences. Analysts query the same tables that data scientists use to train models. Both groups read from a single governed source. The frequent extract, transform, and load processes previously required to move data between the lake and the warehouse are largely eliminated.

Why it matters beyond the infrastructure diagram

Running a data lake and a data warehouse simultaneously creates architectural complexity and raises costs in both money and time. You store the same data in two locations. You must run pipelines to synchronise these copies, and when those pipelines fail, staff must repair them. Analysts and data scientists query different versions of the data, leading to conflicting results. Eventually someone discovers these discrepancies and spends a week identifying the cause.

A unified lakehouse removes most of that administrative work. There is only one copy of the data, one set of access controls, and one location for quality rules. Organisations that consolidate their systems usually cut their deployment cycles by several weeks. The lakehouse architecture does not always accelerate the building process itself. Instead, the main benefit is that engineers no longer spend time moving data between two separate systems. They can focus their time on the specific problems they were hired to address.

The governance angle people underrate

Governance receives less attention than cost. Traditional data lakes do not provide a reliable audit trail because data arrives in various formats without an enforced schema. A lakehouse uses a metadata layer to track every write, schema change, and table version. This lets you answer compliance enquiries about report data by running a query instead of guessing. We have taken part in many audit preparation meetings and understand the value of this capability during high-pressure situations.

A unified architecture lets a business respond to change more quickly because the entire team uses the same source of truth.

How to plan a lakehouse before selecting tools

Successful projects start with an accurate assessment of the existing infrastructure rather than a comparison of different vendors. If you skip this assessment, you will create an architecture that looks effective in presentations but fails to resolve the actual issues your teams face.

Create a map of your existing assets

List every source that provides data for your analytics, reporting, and machine learning now: the operational databases, the SaaS tools with an API, the streaming platforms, and the file shares that are not officially managed. Calculate your total data volume and identify which datasets are growing most quickly, since this impacts your storage and computing expenses significantly. Then, speak with the people who use the data. Ask which queries are slow, which reports are most important, and where they spend time fixing discrepancies in numbers that should match. Typically, you will find data duplicated across three or four systems, with each copy slightly out of date and maintained manually by someone.

Record specific results instead of goals

"Modernise our data platform" is a vague sentiment rather than a specific goal. You should establish measurable objectives instead. For example, you could reduce the time between data ingestion and analyst access from two days to two hours. You could also choose to support fraud detection on live transaction streams. These concrete targets serve two purposes. They guide the architectural decisions you make, and they provide clear evidence of value when stakeholders ask about the results of the investment.

Pick storage and a table format that match your team

Most lakehouses are built on Amazon S3, Azure Data Lake Storage, or Google Cloud Storage, with the choice usually based on existing data location and team expertise. A table format is then selected, with Delta Lake, Apache Iceberg, and Apache Hudi being the most common options. This table format transforms a data lake into a lakehouse by adding features like a transaction log, schema enforcement, and versioning, which are not present in a simple collection of Parquet files. These three formats offer similar capabilities. The final decision usually depends on the processing engine your team already uses and the extent to which you want to use the associated tools.

The five layers that make a lakehouse work

Regardless of the specific vendor, every lakehouse consists of the same five layers. Each layer performs a specific function and passes data to the next one.

If you grasp these factors accurately, you can design a system that satisfies the requirements of your particular workload.

Ingestion layer

Data is ingested into the system at this point. It is gathered from operational databases, APIs, streaming platforms, and file stores, and then loaded into lakehouse storage. The system must handle both batch and real-time flows at the same time. This covers nightly exports from billing databases as well as continuous streams of clickstream events from websites. Teams commonly employ AWS Database Migration Service for relational sources, Kafka for streaming data, and custom connectors for SaaS tools that rely on REST APIs.

Users frequently omit data-quality validation during ingestion, causing issues later. You ought to carry out schema checks, type validation, and business-rule enforcement at this point. This enables identification of malformed records before storage and before analysts use them to create dashboards.

Storage layer

Raw and processed data reside on low-cost object storage. The data is stored in open formats such as Parquet or ORC, enabling any tool to read the files without a proprietary licence. This architecture cuts costs by separating storage from compute. You are charged for processing power only while a query runs, instead of paying for idle servers. To keep performance as data volume reaches terabytes, you arrange the data into partitions that match your query patterns.

Metadata layer

This layer transforms a data lake into a lakehouse. It determines which files belong to particular table versions and enforces ACID transactions to avoid data corruption during concurrent write operations. It also keeps schema definitions to guarantee data consistency. Delta Lake, Apache Iceberg, and Apache Hudi all carry out these functions with different transaction log methods while storing the underlying data in open formats.

It also offers features that a standard data lake lacks. These include time travel, enabling queries of a table as it existed on a specific past date, and zero-copy cloning, which quickly creates a development copy of a dataset. This layer also handles quality rules, lineage tracking for audits, and access control.

API layer

The API layer enables analytics tools and applications to query data without directly accessing the underlying storage system. It translates SQL, DataFrame calls, and REST requests into optimised reads from object storage. This layer also handles caching and query planning to make reading from cloud storage as fast as reading from a local warehouse disk. Business analysts retrieve this data with SQL tools like Tableau or Power BI, while data scientists use Python or Spark to access the same information. Both groups employ different methods to reach the same tables.

Consumption layer

This is where people use the thing: BI dashboards, notebook environments, machine learning frameworks, all reading directly from lakehouse storage through the API and metadata layers below. Nobody is working from a third copy that went stale on Tuesday. You set access permissions and resource quotas here so one analyst running a heavy query at 9am doesn't slow everyone else down, and result caching keeps repeated queries fast without extra engineering.

Common patterns: medallion and lambda

Most lakehouse architectures employ two specific patterns. You should grasp both before designing your own system.

Medallion architecture

The Medallion architecture arranges data into bronze, silver, and gold layers. Each layer denotes a higher level of refinement. The bronze layer holds raw data in its original form. The silver layer holds that same data after it has been cleaned, validated, and joined for general use. The gold layer includes aggregated tables crafted for specific business needs, such as the data used in dashboards. This pattern aids debugging because it lets you trace incorrect numbers back through each stage of the process.

Lambda architecture

Lambda operates a batch pipeline and a streaming pipeline simultaneously rather than selecting one over the other. The streaming path delivers a fast, approximate result in near real time. The batch path executes once daily to incorporate late data and delivers the accurate final result by the next morning. This approach offers both speed and accuracy, although they originate from different pipelines at different times. You should tell the users of these figures about this trade-off.

Most organisations employ the medallion architecture to shape data quality and the lambda architecture to shape timing. These two frameworks integrate easily. The bronze and silver layers operate on the batch path. Once the aggregation logic is defined, the gold layer can serve data quickly enough for near-real-time dashboards.

Lakehouse versus lake versus warehouse

These three architectures differ in costs, governance requirements, and performance levels. You must grasp these trade-offs before selecting an architecture to avoid project failure later.

• Data warehouses deliver fast SQL performance and strong consistency for structured data. However, storage costs are high because storage and compute are coupled. The proprietary data format also makes it difficult and costly to use the same data for machine learning.
• Data lakes lower costs by permitting any file format to reside on inexpensive object storage without premium fees. However, this approach lacks governance. There are no systems to enforce schemas or to prevent simultaneous write operations. Consequently, teams must devote considerable time to validating data quality before using it.
• A lakehouse preserves the storage costs and open formats of a data lake while employing a metadata layer to deliver the ACID transactions and schema enforcement present in a data warehouse. Analysts and data scientists share the same tables. This eliminates most of the ETL work needed to synchronise two separate systems.

None of the three options is invariably the best choice. A small team that mainly uses structured reporting and does not intend to use machine learning may find a simple data warehouse sufficient, and building a lakehouse would be unnecessarily complex. A lakehouse is most valuable when you have diverse data needs, encompassing both structured and unstructured data, business intelligence and machine learning, and both batch and streaming data, all needing access to a single source of truth.

Common causes of failure in lakehouse projects

We observe the same small set of errors occurring frequently. The most common mistake is treating the lakehouse as a technology purchase rather than an architectural decision. Teams often choose Databricks or Snowflake and deploy the platform before deciding which data to include or who is responsible for it. This sequence yields a costly and well-engineered system that lacks organisation.

The second mistake is postponing governance until a problem arises, such as an audit or a security incident. It is cheaper to include Access control and lineage at the start than to add them later to a system already supporting thirty pipelines. The third mistake is underestimating the migration process. Moving from a warehouse to a lakehouse demands months of detailed work on individual tables. Teams that aim for a rapid transition often find themselves mid-migration a year later.

The fourth issue is the one we encounter most frequently during client projects. It involves building the lakehouse before anyone has properly mapped the existing data estate. You cannot design effective partitioning, logical access boundaries, or a functional ingestion strategy for data you have not yet analysed. If you skip this step, gaps in coverage will only surface during an outage or an audit. These are the two most difficult moments to identify such problems.

Getting started

The main point of this guide is that a lakehouse should be built only if your teams need mixed workloads on the same governed data. If a standard data warehouse satisfies your current needs, a lakehouse becomes an unnecessary expense. You need to assess your current infrastructure accurately and define measurable outcomes. Choose your storage and table formats according to the actual skills of your current team instead of the skills you expect them to acquire later.

If you need a clear view of your own gaps before spending on tooling, that is precisely the assessment we conduct. Talk to us and we will tell you directly whether a lakehouse is the appropriate next step for your data, or if the real problem lies elsewhere entirely.

If you are currently tackling this issue rather than researching it, our data consultancy offers these services. We start by demonstrating the specific capabilities of your existing data.

Frequently asked questions