Comment by MrPowers | Hacker News Reader

Comment by MrPowers | Hacker News Reader

Parent Full thread

Great article.

> data engineers have been increasingly adopting software engineering best practices

I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days. I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot.

> The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer.

I think the Lakehouse architecture is the real future of data engineering, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision.

It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).

I am a data engineer, and I STILL don't understand the differences between the following terms:

1. Data Warehouse

2. Datalake

3. Data Lakehouse

4. Data Mesh

Can someone please clearly explain the differences between these concepts?

Let me help you out:

Data Warehouse. SQL engine that is much bigger than the databases of yester-year. Their size and scalability give you capabilities that DBAs of a MS SQL dB wouldn't necessarily allow you to do, but think "wow, that is a big database!"

Data Lake. Now your database is stored in blob storage (which is like CSV files on your computer) - and the blob storage is HUGE (like hundreds of Terabytes or larger). Also, you store unstructured data, too. So: pictures, JSONs, raw HTML. Whatever

Data Lakehouse... do you mean Delta Lake? This is a made up word from Dara Bricks. Take your Data lake, and slap a SQL engine on top of it, and add in some marketing slang ;)

Data Mesh. THIS is more about organizations than infrastructure. Imagine a Data Warehouse or Datalake. Now find 5 key stakeholders around your company. Let each of them have a playground in your warehouse. You drop the data into one zone, then they copy it, transform it, and share it in their zone - but they have to play by your rules. Bam. Data Mesh

The Delta Lake is a marketing term from Databricks to, in part, market their Delta file format and all the clusters you’ll be spinning up.

Delta files are actually an amazing format though. At their most basic, they took Parquet files (columnar database) and let you stream off them really easily. Which takes a lot of complexity out of your pipelines - dont need Kafka for everything, don’t need to figure out when new rows get added (or a whole other set of jobs around that).

But using Delta files really can change the way you develop pipelines (and ML pipelines), so I forgive them for inventing a new term.

As someone that is familiar with what a Lake House Arch is, I remain confused with what Delta Lake is mostly bc I find it difficult to differentiate what is Databricks Delta Lake marketing and the virtues of the Delta Lake Arch (Delta files, etc). It's frustrating, and I've given up...

I am however keeping up to date with Apache Iceberg bc it's much easier to follow and it seems to have a lot of advantages over the Delta Lake external table format (delta files?).

Iceberg seems to be better especially in handling schema evolution and drift. They both seem to use parquet and avro below the surface and generally have the same design, but am I missing anything by dismissing and ignoring Delta Lake?

Disclaimer: I work for Databricks.

With that out of the way, the Data Lakehouse name has a Marketing motivation, but the concept is “let’s not have a DW and a Lake as separate structures based on different technical stacks - let’s have one stack that can do both”. That is a very valid proposition, because the information needs tend to span what a DW or a Lake can easily do by themselves. Also, having one common stack makes it much easier to manage all data and generally be productive.

If you look back even at Inmon’s Corporate Information Factory, you’ll see mentions to unstructured data coming in, and the expectation that all data, structure or unstructured, should become useful information for some business purpose. This was back in the 90s, the need was already there, but the technology was not.

The Lakehouse as proposed by Databricks is essentially a combination of technologies (including but not limited to Delta Lake format) to deliver that unified design. It’s a platform where you can process all kinds of data, apply structuring where needed, manage quality, do DataOps and generally deliver information, regardless of the structure or lack of it in the original data.

So, to put it simply, a Lakehouse is a platform (or technical stack, if you want to build your own) where you can implement a DW, a Data Lake and mix capabilities of both to deliver data products more efficiently. It’s more than just the union of DW and Lake, because the combined capabilities allow you to do things that neither could by itself. For instance, parse a large set of documents for sentiment and context to generate metrics on some business topic. Environment, Sustainability and Governance - ESG - is one example area where the results are BI-like metrics for reports and dashboards, but the sources are usually text documents. Social media processing for Marketing purposes is another.

Thank you, excellent explanation.

Marketing and hype. Especially data mesh. I gave up listening to talks/reading papers about it – could never wrap my head around the "mess"

Warehouse and Lake also feel mostly like distinctions without difference, though if you really want, I think this is the technical difference: - warehouse: structured data for analytics, outputted from ETL jobs - lake: a dump for unstructured data, to be cleaned up later: EL, then possibly T

Moreover, people want to push this even further, with "data river" - streaming(-like) data that is "continuously" transformed (isn't that just an ETL data pipeline?)

1. Data Warehouse

Recommend reading up on Kimball vs Inmon.

Kimball -- centralised storage in a "dimensional model". Uses a star/snowflake schema where the central FACT table (containing the raw transactional data) can be filtered and aggregated based on choices made in dimension tables (edges of the star/snowflake).

    demographic  account_types      product_types
       \        /                    /
        \      /                    / 
        customer                  product
          \                      /
           \                    /
       FACT table (sale $, order quantity, order ID, product ID)

customer, account_types etc are dimensions to filter your low-level transactional data. The schema looks like a snowflake when you add enough dimensions, hence the name.

The FACT table makes "measures" available to the user. Example: Count of Orders. These are based on the values in the FACT table (your big table of IDs that link to dimensions and low-level transactional data).

You can then slice and dice your count of orders by fields in the dimensions.

You could then add Sum of Sale ($) as an additional measure. "Abstract" measures like Average Sale ($) per Order can also be added in the OLTP backend engine.

End users will often be using Excel or Tableau to create their own dashboards / graphs / reports. This pattern makes sense in that case --> user can explore the heavily structured business data according to all the pre-existing business rules.

Pros:

- Great for enterprise businesses with existing application databases

- Highly structured and transaction support (ACID compliance)

- Ease of use for end business user (create a new pivot table in Excel)

- Easy to query (basically a bunch of SQL queries)

- Encapsulates all your business rules in one place -- a.k.a. single source of truth.

Cons

- Massive start up cost (have to work out the schema before you even write any code)

- Slow to change (imagine if the raw transaction amounts suddenly changed to £ after a certain date!)

- Massive nightly ETL jobs (these break fairly often)

- Usually proprietary tooling / storage (think MS SQL Server)

---

2. Data Lake

Throw everything into an S3 bucket. Database table? Throw it into the S3 bucket. Image data? Throw it into the S3 bucket. Kitchen sink? Throw it into the S3 bucket.

Process your data when you're ready to process it. Read in your data from S3, process it, write back to S3 as an "output file" for downstream consumption.

Pros:

- Easy to set up

- Fairly simple and standardised i/o (S3 apis work with pandas and pyspark dataframes etc)

- Can store data remotely until ready to process it

- Highly flexible as mostly unstructured (create new S3 keys -- a.k.a. directories -- on the fly )

- Cheap storage

Cons:

- Doesn't scale -- turns into a "data swamp"

- Not always ACID compliant (looking at you Digital Ocean)

- Very easy to duplicate data

---

3. Data Lakehouse

Essentially a data lake with some additional bits.

EDIT: aims to mitigate against the "data swamp" problem, improve flexibility / performance when changing how some-random-column X is calculated, improved governance and auditability.

A. Delta Lake Storage Format a.k.a. Delta Tables

https://delta.io

EDIT: apparently there are a few more players and not just delta lake (Apache iceberg). But there’s still questions around how open the open source for these things are.

Versioned files acting like versioned tables. Writing to a file will create a new version of the file, with previous versions stored for a set number of updates. Appending to the file creates a new version of the file in the same way (e.g. add a new order streamed in from the ordering application).

Every file -- a.k.a. delta table -- becomes ACID compliant. You can rollback the table to last week and replay e.g. because change X caused bug Y to happen.

AWS does allow you do this, but it was a right ol' pain in the arse whenever I had to deal with massively partitioned parquet files. Delta Lake makes versioning the outputs much easier and it is much easier to rollback.

B. Data Storage Layout

Enforce a schema based on processing stages to get some performance & data governance benefits.

Example processing stage schema: DATA IN -> EXTRACT -> TRANSFORM -> AGGREGATE -> REPORTABLE

Or the "medallion" schema: Bronze -> Silver -> Gold.

Write out the data at each processing stage to a delta lake table/file. You can now query 5x data sources instead of 2x. The table's rarity indicates the degree of "data enrichment" you have performed -- i.e. how useful have you made the data. Want to update the codebase for the AGGREGATE stage? Just rerun from the TRANSFORM table (rather than run it all from scratch). This also acts as a caching layer. In a Data Warehouse, the entire query needs to be run from scratch each time you change a field. Here, you could just deliver the REPORTABLE tables as artefacts whenever you change them.

C. "Metadata" Tracking

See AWS Glue Data Catalog.

Index files that match a specific S3 key pattern and/or file format and/or AWS S3 tag etc. throughout your S3 bucket. Store the results in a publicly accessible table. Now you can perform SQL queries against the metadata of your data. Want to find that file you were working on last week? Run a query based on last modified time. Want to find files that contain a specific column name? Run a query based on column names.

EDIT: also helps with data governance and auditability as all of your files/tables are tracked in a centralised location.

Pros:

- transactional versioning -- ACID compliance and the ability to rollback data over time (I accidentally deleted an entire column of data / calculated VAT wrong yesterday)

- processing-stage schema storage layout acts as a caching layer (only process from the stage where you need to)

- no need for humans to remember the specific path to the files they were working on as files are all indexed

- less chance of creating a "data swamp"

- changes become easier to audit as you can track the changes between versions

Cons:

- Delta lake table format is only really available with Apache Spark / Databricks processing engines (mostly, for now)

- Requires enforcement of the processing-stage schema (your data scientists will just ignore you when you request they start using it)

- More setup cost than a simple data lake

- Basically a move back towards proprietary tooling (some FOSS libs are starting to pop up for delta tables, plus git-based data versioning is a thing)

---

4. Data Mesh

geoduck14's answer on this was pretty good. basically have a data infrastructure team, and them domain-specific teams that spring up as needed (like an infra team looking after your k8s clusters, and application teams that use the clusters). domain specific data team use the data platform provided by the data infrastructure team.

Previously worked somewhere in a "product" team which basically performed this function. They just didn't call it a "data mesh".

Very thorough answer on the DW and Data Lake points. On the Lakehouse, though, as I wrote above, it’s not just the data storage format (Delta, Iceberg, whatever). The storage is important, because it creates the possibility of using cloud storage as a basis for table-like structures, with ACID (or a subset of it) capabilities. Beyond that, you still need management, performance, security, SQL access and probably some more that I forget.

The goal for a Lakehouse is to create a platform/stack that can effectively do everything a DW does, everything a Data Lake does (so, all your pros above), get rid of most of the cons and also allow mixing and matching of the functionalities. Acknowledging this need to handle all kinds of data is not new; having the technology to deliver on it is, and that’s what the Lakehouse is.

Just seen this response. Thanks for the comment.

I was on the fence about including mentions of querying mechanisms like AWS Athena and the like, but you’re right it’s all part of the same horse (or house).

Was 5am by the time I finished writing this though… so I kind of had to stop (hence the very short data mesh section!).

data fabric?

You mean Lake 'Shanty' Architecture (think DataSwamp vs DataLake) am I right?

But in all seriousness, I totally agree with your opinion on LakeHouse Architecture and am especially excited about Apache Iceberg (external table format) and the support and attention it's getting.

Although I don't think that selecting any of these data technologies/philosophies comes down to making a mutually exclusive decision. In my opinion, they either build on or compliment each other quite nicely.

For those that are interested, here are my descriptions of each...

Data Lake Arch - all of your data is stored on blob-storage (S3, etc) in a way that is partitioned thoughtfully and easily accessible, along with a meta-index/catalogue of what data is there, and where it is.

Lake House Arch - similar to a DataLake but data is structured and mutable, and hopefully allows for transactions/atomic-ops, schema evolution/drift, time-travel/rollback, so on... Ideally all of the properties that you usually assume to get with any sort of OLAP (maybe even OLTP) DB table. But the most important property in my opinion is that the table is accessible through any compatible compute/query engine/layer. Separating storage and compute has revolutionized the Data Warehouse as we know it, and this is the next iteration of this movement in my opinion.

Data Mesh/Grid Arch - designing how the data moves from a source all the way through each and every target while storing/capturing this information in an accessible catalogue/meta-database even as things transpire and change. As a result it provides data lineage and provenance, potentially labeling/tagging, inventory, data-dictionary-like information etc... This one is the most ambiguous and maybe most difficult to describe and probably design/implement, and to be honest I've never seen a real-life working example. I do think this functionality is a critical missing piece of the data stack, whether the solution is a Data Mesh/Grid or something else. Data Engineers have their work cutout on this one, mostly bc this is where their paths cross with those of Application/Service Developers, Software Engineers. In my opinion, developers are usually creating services/applications that are glorified CRUD wrappers around some kind of operational/transactional data store like MySQL, Postgres, Mongo, etc. Analytics, reporting, retention, volume, etc are usually an after thought and not their problem. Until someone hooks the operational data store up to their SQL IDE or Tableau/Looker and takes down prod. Then along comes the data engineer to come up with yet another ETL/ELT to get the data out of the operational data store and into a data warehouse so that reports and analytics can be run without taking prod down again.

Data Warehouse (modern) - Massive Parallel Processing (MPP) over detached/separated columnar (for now) data. Some Data Warehouses are already somewhat compatible with Data Lakes since they can use their MPP compute to index and access external tables. Some are already planning to be even more Lake House compatible by not only leveraging their own MPP compute against externally managed tables (eg), but also managing external tables in the first place. That includes managing schemas and running all of the DDLs (CREATE, ALTER, DROP, etc) as well as DQLs (SELECT) and DMLs (MERGE, INSERT, UPDATE, DELETE, ...). Querying data across native DB tables, external tables (potentially from multiple Lake Houses, Data Lakes) all becomes possible with a join in a SQL statement. Additionally this allows for all kinds of governance related functionality as well. Masking, row/column level security, logging, auditing, so on.

As you might be able to tell from this post (and my post history) is that I'm a big fan of Snowflake. I'm excited for Snowflake managed Iceberg tables and then consume the data with a different compute/query engine. Snowflake (or other modern DW) could prepare the data (ETL/calc/crunch/etc) and then manage (DDL & DML) it in an Iceberg table. Then something like DuckDB could consume the Iceberg table schema and listen for table changes (oplog?), and then read/query the data performing last-mile analytics (pagination, order, filter, aggs, etc).

DuckDB doesn't support Apache Iceberg, but it can read parquet files which are used internally in Iceberg. Obviously supporting external tables is far more complex than just reading a parquet file, but I don't see why this isn't in their future. DuckDB guys, I know you're out there reading this :)

https://iceberg.apache.org/

https://www.snowflake.com/guides/olap-vs-oltp

https://www.snowflake.com/blog/iceberg-tables-powering-open-...

Finally one of my favorite articles:

https://engineering.linkedin.com/distributed-systems/log-wha...

Great write-up. I would add that I actually have seen something like a "Data Mesh" architecture, at a bank of all places. The key was a very stable, solid infrastructure and dev platform, as well as a custom Python library that worked across that Platform which was capable of ELT across all supported datastores and would properly log/annotate/catalog the data flows. Such a thing is really only possible when the platform is actually very stable and devs are somewhat forced to use the library.

I'm going to use "Lake Shanty" in the future. Powerful phrase to describe what happens when you run aground on the shore of a data swamp.

> It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc.

What are some good resources that can help educate folks on these differences?

Short version:

- data warehouse: schema on write. you have to know the end form before you load it. breaks every time upstream changes (a lot, in this world)

- data lake: schema on read. load everything into S3 and deal with it later. Mongo for data platforms

- data lakehouse: something in between. store everything loosely like a lake, but have in-lakehouse processes present user-friendly transforms or views like a warehouse. Made possible by cheap storage (parquet on S3), reduces schema breakage by keeping both sides of the T in the same place

Materialised views for cloud storage?

I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg

Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.

(OP's coworker) We actually published a guide on data lakes/lakehouses last month! https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b...

covering:

- What’s a Data Lake and Why Do You Need One?

- What’s the Differences between a Data Lake, Data Warehouse, and Data Lakehouse

- Components of a Data Lake

- Data Lake Trends in the Market

- How to Turn Your Data Lake into a Lakehouse

"Lakehouse" usually means a data lake (bunch of files in object storage with some arbitrary structure) that has an open source "table format" making it act like a database. E.g. using Iceberg or Delta Lake to handle deletes, transactions, concurrency control on top of parquet (the "file format").

The advantage is that various query engines will make it quack like a database, but you have a completely open interop layer that will let any combination of query engines (or just SDKs that implement the table format, or whatever) coexist. And in addition, you can feel good about "owning" your data and not being overtly locked in to Snowflake or Databricks.

Reply on news.ycombinator.com