Hacker News Reader

Comment by teej | Hacker News Reader

The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates). This makes it orders of magnitude easier to diagnose, debug, and assert the correctness of data.

So these trends aren’t in a vacuum, they directly support the issues you discuss.

> Most of the projects I've seen or heard of are just people taking the same old data ...

I don’t disagree with you here. But in my experience it’s about getting Frank in marketing to use the same numbers as everyone else.

When you have 5 different ads platforms that all take revenue credit for a single conversion and have conflicting attribution models, and none of them add up to what accounting says is in the bank account. That’s a hairy problem.

There are different flavors of that class of problem at lots of companies.

View on HN

> The biggest shift has been towards data lake (store everything) away from data cubes (store aggregates).

I don't think this is any shift. The "store everything" has always existed in my experience, that's how the aggregates were built in the first place. The aggregates were for speed and convenience, and you drill-down as necessary, including to the individual record level.

Maybe the shift is people thinking that it's cheaper to just analyze the entire corpus on-demand because we can throw a spark cluster at it?

I agree, data warehouse was what the data lake is today. Data cube is the aggregation of data in the warehouse, and then you can drill down and roll up. Difference between warehouse and lake is the emphasis on correctness (one canonical data model) and deduplication of data (when warehouses were invented, storage was expensive so one tried to normalise it into a star schema with as little duplication as possible — when emergence of cheap storage, this is less important and we can spend less time developing fancy ETL processes to make everything fit into one, conformant data model).

> this is less important and we can spend less time developing fancy ETL processes to make everything fit into one, conformant data model

And that's precisely why modern data processes are inferior to 20 years ago. People reinvent the wheel over and over and spend massive budgets on unnecessary tech stacks that would be alleviated if the time was simply taken to model the data.

A clean data model is about a whole lot more than simply storage space.

Agree if people would model everything to a “Grand Unified Data Model” of everything, it would be a lot more efficient... unfortunately that is very hard and puts a massive bureaucracy around data governance and management. It slows things down. I guess a more modern approach is to relax those constraints a bit and realise that some data can be expressed in different ways, and that duplication isn’t too much of a concern because storage is cheap. That said, it’s not an excuse for reinventing the stack over an over. I think the thirst to reinvent has largely been driven by the shift from expensive proprietary solutions like Teradata and Oracle to open source ones. That’s a positive shift.

And this is why big enterprises are moving out of data warehouse model for processing all the data and prefer a data lake concept. It is better to centralize some aspects of the data, but definitely not all (like common relational model for all sources, as it is in data warehouse). The data warehouse model quickly becomes super expensive to maintain and evolve.

You miss the critical difference -- nowadays people don't store aggregates, they just scan sharded data very fast. That simplifies a lot of things, because you don't need to keep two databases in sync (raw and aggregated).

I was working in data analytics + data science a decade ago and we stored everything, not aggregates, and pushed them through hadoop. I have been "out of the game" since then. What has changed that is making people saying "store everything" is a new phenomenon? (genuine question bc I am clearly missing something.)

It’s not a new phenomenon so much as it has emerged as an important shift from the status quo 20 years ago.

What’s changed in the last 10 years are the access patterns. There’s increased demand to have arbitrary query access over the raw data. The most impactful technology changes have been about pushing the access layer (queries, stream & batch processing, dashboards, BI tools, etc) down as close to the raw data as possible and making that performant. What’s fallen out of that are better MPP OLAP databases (snowflake), new columnar formats (parquet), SQL as the transform layer (dbt).

Ah that makes sense. Thanks.

why is it actually that SQL "re-emerged" as the transformation layer? I thought that it first shifted from SQL to Query Builders inside talend, matillion etc. Why now SQL again?

Probably just the emergence of dbt? I’ve only been doing ETL for a couple years personally but couldn’t imagine using so much SQL in our pipelines without a framework like dbt

The problem of data confusion you describe is resolved by replacing management. That’s not an engineering issue that requires new technology (consider the source of social power for author; selling technology).

That’s an engineering issue that needs new engineering management who don’t enable wasting company resources making incompatible APIs in the first place.

We already did the monolithic DB design, I used to name those hosts “ocean”. And we already know the math. “Data lake” is just more jargon by a salesman to obfuscate peddling the same old abstraction, and wow fresh grads with new words for hyping the same old habits.

While not the author of this piece, Bezos is quoted as pointing out how circular social behavior is.

What do you think the odds of this author being on a similar page?

Have humans evolved much in 100 years? Or does the con simply get rewritten for the next generation to hide a simple truth?

What’s keeping people going in this circle isn’t logistical necessity. It’s us.

Reply on news.ycombinator.com