DataLake
martinfowler.com
martinfowler.com
The data lake is a great idea for researchy or data-sciencey shops where the analysis is the bread and butter. What happens here is that people get data, clean it themselves in ways only they care about, and then go about their business.
The problem is there is no one centralized repository of good, clean datasets. Instead it's spread across people, who (unwittingly) hoard domain knowledge on them, and then are now stuck with doing anything else that's required with it, as well as passing it off to other people. Then these people leave. So people keep repeating work in a non-DRY way. Or they do clean it for everyone else to use, but the format and naming isn't standard across datasets, or is spread across weird locations, which again complicates the workflow.
Instead, what we want is an internal tool to be able to search and browse data (think data.gov or CKAN), to have a native command to do that from within their tool of choice (R, STATA, matlab), and to have a process that goes with this.
The hope for the end result is a streamlined process where when people wonder if something is viable or interesting, they can find a load a dataset within seconds and test a theory, as opposed to having to ignore the impulse because it's too much of a PITA.
Not much is public about it but it is an interesting approach.
Will try to write more about what I recall from public talks.
What I meant with techincal performance is crap: federated joins between databases inside their infra can be very very slow. e.g. some SPARQL queries of theirs run for a week or so. However, this is not a problem, as any other solution would cost them many more man weeks and billable hours to get the same answer.
Datawarehousing is not the answer they already had 70+ datawarehouses and one more was not going to be the solution. They needed a way to have the datawarehouses talk to each other.
If I recall the story correctly, they went down this road when one of their unpatented drugs was being tested by a 3rd party company that went bankrupt. Legal determined they needed to get all the Eli IP back in house before the bankruptcy procedure started. The legal team went in burnt many many many billable hours and found 2 compounds that they needed to retrieve. The SPARQL team testing stuff out found 2 more over the weekend, with about 2 hours of development work.
Since then they have invested a lot in SPARQL as well as security and caching layers to make SPARQL work as a lingua franca between all their databases.
Serious question, does anyone know any companies that are employing this successfully? And if so, in what fashion? I'd definitely love to hear about a success story and what value was provided.
Edit: The example in the article seems to be more related to the failure of the data warehouse than the success of the data lake.
I am working on a "Google for SEC filings". There are about 15 million company filings and other data spanning 20 years available on sec.gov. The data is about 700 GB compressed, but unfortunately you have to download each filing individually from their FTP server. When I first started, I wrote a script that would download the filing and then process it into the format I wanted. However their FTP server is very slow and there are 15 million individual downloads, so it was taking forever. Rather I wrote a script that mirrored the FTP server to S3 as fast as possible while still being respectful to their bandwidth and server capacity. And this still took almost 3 weeks.
Now I have a "data lake" of raw SEC filings and other data which I can pull from at any time on S3. And the important part is the performance is significantly better so the processing time is relatively small.
one-time upload cost (bandwidth free): (15 million / 1000) POSTs * $0.005 = $75
one-time download cost: ((15 million / 10000) GETs * $0.004) + (700 GB * $0.09) = $69
I see this same strategy being attempted at some non-tech companies as well. Too early to say whether it will succeed.
The main benefit as I see is when data sources are external, with ill-defined or ambiguous schemas. Often when you fit data into an ETL pipeline, you find out issues at the output of the pipeline, but the fixes need to happen way upstream. Often this involves rebooting the entire process entirely and renormalizing all your data somehow.
If you delay interpretation and normalization to later in the processing pipeline (i.e. in the system I worked on, we did it lazily at interpretation time), then doing smarter things with the data is a matter of changing code -- and it's a lot easier to ship fixes to code than to ship fixes to data!
The usual problems with these are:
(1) getting hold of external clean up functionality. Sometimes it was stored with the database itself as stored procedures, but sometimes it was not. Remedy: make sure clean up functionality is readily available.
(2) dealing with new kinds of "dirt" (i.e. problematic data in the data lake). Remedy: this is unavoidable. The best you can do is to insert lots and lots safety checks and diagnostics into your clean up programs
In the system I designed, I have dozens of data sources feeding in the same "type" of data, but all in different formats and terminology. My multi-stage system applies transformations, including business logic and data cleansing, to each source individually. Then the data sources are combined and linked to other information sources as a unified data model. Consumer applications [1] can take the unified data model and quickly get up and running with for similar analytic applications that maybe just need different views. Or more customized applications can take the transformed data of a single source, apply additional logic that incorporates the raw data, and then incorporate other information sources. In this way, the system is flexible and open, while providing solid data governance.
Overview:
(1) Data Source -> (2) Data Source Transformation -> (3) Unified Data Model -> Dashboard
A consumer application could acquire data from (1), (2), (3), and/or other data sources.
[1]: In this context, "consumer application" means an data transformation process or a data presentation application, not anything to do with the B2C market sector.
Here in Spain some private banks are making heavy use of those technologies to replace their reporting originally based on mainframe technology.
Perhaps a more business analyst oriented concept of the data lake may be the semantic layer [1]. This concept may differ from Fowler's in that is not so data oriented, and augments it, but underneath, some of the goals, as providing self service querying facilities to analysts, and making use of as much of the ingested data as possible, are similar.
[1] https://www.veroanalytics.com/blog/its-time-to-unleash-the-s...
- All data is CSV or json-document-per-row text files on Amazon S3.
- We have a Django web application that keeps track of metadata (which dataset lives in which S3 bucket/folder, who uploaded it, the names of the columns in that dataset, and their types).
- The REST API in the Django web piece can provide temporary signed S3 URLs that can allow anyone in the company to create a new dataset and upload the files to S3.
- The REST API also provides Hive and Redshift "CREATE TABLE" commands for all datasets (which it builds from the columns/types data stored in the DB).
We've talked several times about open sourcing it but haven't gotten around to making that happen yet.
> The data lake is schemaless, it's up to the source systems to decide what schema to use and for consumers to work out how to deal with the resulting chaos. Furthermore the source systems are free to change their inflow data schemas at will, and again the consumers have to cope. Obviously we prefer such changes to be as minimally disruptive as possible, but scientists prefer messy data to losing data.
So basically, a data lake is a fancy word for a file system?
Hadoop and NoSQL systems are critical for this role since often it is extremely time consuming to (a) design the final end state schema and (b) create the ETL processes to populate it. So the idea is to just fill the data lake as quickly as possible and then work out later how to use it.
Data lakes are a concept that apply mainly to enterprises so we are talking about big data and complex, multi disciplinary/functional schemas.
To me, it looks more like an ODS (Operational Data Store) with some data-mart like curation steps before it hits users.
A data lake should be storing the data in it's original form, without transformation.
Yes, I get that. I was probably oversimplifying.
Practically speaking, the BI projects I've been on for small to medium enterprises (<150M in size, non-tech) still have predominantly relational data sources.
In many of those cases, a warehouse is an expensive effort in over-engineering, but a data-lake-like ODS (without transformations in the main data tables) with curated subject-area marts is a more sensible solution.
--edit, added "non-tech" to my qualifier
Well, when you get down to it, any database is a just a file system with an API :)
I think a "data lake" might be a combination of raw files with metadata. For example, raw scrapes on S3 + an entry on Postgres with a link to the file, the date it was scraped and the version of the bot that scraped it.
This makes me curious, what are the costs down the road VS upfront costs? There's a side of me that just feels like it's somewhat laziness to not employ a schema -- even a very flexible one. And if data scientists were to come together and agree upon some flexible schema, would it not be at least a step in the right direction and one capable of constant iteration / improvement?
I am of the mindset that single-responsibility-principle applies here. If you could build a framework that pipes out some data in a well-tested, reliable manner, then all analysts could hook up to that single pipe -- not create their own hoses and fishing rods.
The only purpose I see in a data lake is to provide a single access point to all the stores in the organization. Since it is up to the consumers to digest this data, you have to imagine this will sprout many different solutions unless the different consumers work together to create a framework to pull out of the lake. If they did do that however, I have to imagine you'd be on your way to building a schema because it would be easier for the framework to interface with.
We're working on something adjacent at Silota. We pull in your CRM, behavioral analytics and support data and provide an easy to comprehend view layer atop. Our users are not analysts or data scientist, but account managers (more popularly known as Customer Success Managers.)
(and hence the appeal of hosted solutions.)
- assume processes that dump into the lake also dump elsewhere, for backup purposes
- assume that some other process pulls from the lake (and possibly orders or sanitizes the data) and then backs it up
?
Really counting on other people/teams to have backups of your data is asking for pain.
I really like that it will integrate with a large number of data analysis products... Data Warehousing, Hadoop, Statistical Packages, R, etc. all on one platform/infrastructure would be a huge win to me.
https://www.youtube.com/watch?v=weRCHo36MeM
Now back to your regular programming.