Hacker News Reader

Comment by alexchamberlain | Hacker News Reader

I don't really understand the concept of Data Lake, and wikipedia isn't helping much... is it just a buzzword for a collection of data stores?

View on HN

It's basically a reliable, horizontally scalable object store + a collection of data storage and processing engines.

So amongst the cloud providers, AWS calls a combination of S3 + Glue + Athena (for example) a "data lake", where S3 is the object storage which can store data in various formats, and Glue and Athena are used to transform/process/query the data. See a more detailed article/guide here: https://aws.amazon.com/lake-formation/

If you didn't want to put anything into the cloud and keep all your services on-premise, a local Hadoop cluster could be a data lake, for example using HDFS + Zookeeper + YARN + Hive.

[This is a huge over-smiplification because it's late and I really should be going home :)]

To storage and compute I would also add metadata, this is why most data lake projects end up in data swamps.

When people just dump data in their storage they end having really hard time sharing them in their organization.

I am personally inclined to lump this in with the other enterprisey buzzwords that are generally safe to avoid. From my perspective, the 'data lake' is the actual collection of business systems and your ability to extract useful information from them.

Trying to come up with some unified standard or common API for the extraction, transformation and implementation of useful data from a heterogenous collection of systems sounds like a problematic task at best.

The other, more likely, interpretation of 'data lake' is that it is the staging ground between your ability to do the above stated activity and other downstream systems interested in the data. If the idea is that you are creating the actual normalization layer, I feel like this is still more in the realm of SQL/ETL, as there really isn't any other direction to move that would reduce your entropy in a valuable way (relative to your time invested).

SQLite or Postgres is usually the right choice. This simple rule can help you avoid a lot of pain. Once you have convinced everyone that Postgres is to be used, the only other real barriers are your ability to get a data transport to each business system and the authoring of some SQL scripts. The workload of building a SQL representation of any particular business system is fairly predictable once you break it down to entity-relationship abstractions. Also, using a language with powerful class/object, serialization and database support such as C# or Java can cut your workload by orders of magnitude if you choose a SQL architecture. In C# for instance, you can just write POCOs and use Entity Framework to build out all of the SQL for you. This is not the most performant option by a long shot, but it can get you going incredibly quickly on a first iteration.

> Once you have convinced everyone that Postgres is to be used

... the data has already disappeared.

Systems get shut down and replaced. Operational systems may discard history.

By the time you get a fully operational data warehouse set up, it may be too late to preserve the data.

https://martinfowler.com/bliki/DataLake.html

Awesome link!

The key line for me:

"The data lake stores raw data, in whatever form the data source provides."

The emphasis on"raw" was his, not mine.

Thanks from me, too :-)

Going through the process of defining and planning to build a data lake with my team right now.

A data lake is like you said a collection of data stores, and the industry as a whole hasn't defined it very well past that.

IMHO - A (useful) data lake is a platform that can support any type of data store in any format (be it relational, flat, graph, document, etc) and offer a way to consistently query it. A (useful) data lake does much in the way of managing metadata about those data stores that makes it easy to consume.

I have heard these promises of a single query language for different data formats many times and the results (foisted upon developers and customers) have always far more painful than just sticking to SQL.

To be fair, most data stores have an SQL layer available, even if it's not optimal

Plug: We help discover, catalog and search for metadata and lineage in your data lake. We support all major hadoop distros and cloud platforms. Check us out! https://www.waterlinedata.com

AFAIK, the data lake is the next step in the evolution of the data warehouse. Instead of storing data in a data mart/data warehouse, the concept of the datalake (as a design pattern) is that you don't schemas (support for unstructured data), better support for auditing and data governance/democratization, and schema (?) evolution

I wouldn't say you don't have schema's, rather you have schema-on-read instead of schema-on-write, and you use an extract-load-transform pattern instead of extract-transform-load. The data is replicated as-is into the data lake and only then do you figure out what to do with it.

Yes, in my mind this is the key of a data lake. Take all your raw data and store it somewhere, then provide ways for people to access and query the raw data.

This means ingestion is faster (no transformation) and you don't throw away any data that you might want later. If multiple teams want to query the same data in different ways they have the ability to do so. And ideally it prevents data silos because everyone can stuff their raw data into a master data lake and each team has access to all the data but is responsible for doing the work to make it look like they want.

Reality of the above obviously doesn't always match the theory but schema-on-read/ELT are the easiest ways to handle the above. Typically this involves some kind of Hadoop-style technology, like Hive or SparkSQL for SQL-based querying, Spark for non-SQL, etc. But you've always got the raw data and can go back and re-ELT it from the data lake if your needs change.

I don't know much about the strict definition, but that's how I use them. I have had several clients that want to analyze data they didn't capture in their schema. I'd say: disk is cheap. Throw everything in there (medical records, events, etc.). If we need it later, we'll fish it out. Ugly, but simple.

Note that a data lake does not necessarily replace the data warehouse, but rather often complements it. As such, you store your raw data from various sources in a centralised data store (Hadoop-like, NoSQL, etc.). From there, you prune, clean, select, and potentially aggregate data that you would like to provide in a quality-controlled way to your business users, in a data warehouse. This data warehouse most often will be a more traditional relational data store (usually some flavour of SQL database), which allows users to select data from a curated, pre-selected slice of the overall data stored in the data lake, and which enables easier integration with common reporting tools, whether more traditional standard reporting tools or self-service BI tools.

Practical definitions:

Data lake = place to store unstructured raw data. Usually as files in an object store or Hadoop/HDFS cluster. Analyse with data processing/SQL frameworks. Schema may be part of the data (parquet, avro) or on-read (raw csvs or json modeled into tables).

Data warehouse = place to store processed (semi)structured data. Usually in a distributed columnstore database. Analyze with SQL. Schema is pre-defined by the tables. Usually for smaller, faster, real-time queries or as a cache in-front of a data lake.

Some cloud data warehouses like BigQuery and Snowflake can also query unstructured files and even run on top of the object store so the boundaries are getting blurred. Will probably converge at some point in the future.

The number of definitions here suggest its not well defined, one more :)

A data lake is when you store your data in object storage (S3, GCS etc) as opposed to a filesystem (HDFS) or some indexed datastore (Redshift etc).

This potentially saves a lot of money because you can scale compute separately from storage, and object storage can be extremely cost effective compared to running a distributed filesystem.

Where the two overlap is when you store something like parquet in object storage, the file format is somewhat indexed already so you spend a bit more money preprocessing it but save a lot of money querying it.

I think whether its "raw" json or log files or preprocessed parquet doesnt really differentiate whether its a data lake or not

But interestingly delta.io only supports HDFS currently.

It is a fancy name for a data swamp. :) On the more serious note data warehouses called nowadays data lakes, not sure wjy people prefer this term, maybe because some folks like to think that it is ok to just throw all data in one place and it will be ok to query it like that. I still prefer data warehouse over it.

Some call it "data dump" too.

It kind of makes sense for a cloud to form above a lake.

On another tangent, I wonder if it would be possible to make an un-warrantable cloud? Would it be feasible to create a distributed cloud within the borders of the US or another industrialized country which doesn't actually exist at a particular address?

Reply on news.ycombinator.com