Machine Learning Engineer Guide: Feature Store vs. Data Warehouse | Hacker News Reader

Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

logicalclocks.com

166 pointsnathaliaariza56 comments

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.

Having one central place to find and access data, even if its messy, can be better for understanding and using the data than each business unit/team/... having their own different place for (still messy) data. It can of course also be utterly pointless if done badly, not maintained, not used, ...

Nah, it's a messy pile of data that you want to understand rather than throw away.

You are not wrong on the buzzword, but you are not absolutely right if you suggest that it is merely that; this is a recurring question and interrogation in that specific area.

The fact that it happens to be kinda buzz worthy is a collateral aspect: everything that answers what some people wonder and that is not yet answered plainly, is.

(and I mean, the first on the front page at this very second has : "We hacked apple" in the title.)

The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data.

It's not so much about data itself as an attempt to solve a communications and coordination organizational problem; you decouple sources of data and consumers of data (not the technical systems/databases, but the people and organizational units) to a 'hub-and-spoke' model where the providers of data just supply raw data without getting into a multinational project that takes a year just to identify the potential stakeholders for that data throughout a distributed organization with tens of thousands of employees.

Yea, data lakes are a tech solution to an org problem. Good or bad, it’s what it is.

Done right, they are awesome though. As a DS you can iterate a lot faster if you don't have to access multiple different stores for features or data.

With the caveat that "data lake" means different things to different people, I think it's important to have a repository where the raw data exists un-manipulated in the form that it was ingested.

From the end-user standpoint that's not very useful. But that's why you have data marts that normalize the raw data into a standardized format. Ultimately though the raw data needs to remain the single source of truth. If you skip that step, and only store the post-normalized format, you're likely to run into problems down the road. This could either be because you want to change the normalization format. Or you want to use some aspect of the data that isn't captured in the normalized form. Or even you discover a pre-existing bug that affected all the previous post-ETL data.

> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.

That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."

More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..

As another poster said, if it's the raw data, the raw deal, it's the bets source and nice to have (but could become expensive, so better it is paid for in scale).

If it's something else, it's the antithesis of data science.

> we can still get signal out of it

Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.

That's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).

I say this as someone who built data infrastructure before and after the invention of data lakes, and I have done it every way - the old and the new.. For the record, for a lot of scenarios, the "old ways" actually do still work fine. But there are new opportunities/possibilities too.

I really understand what you are saying... I know the hype problem, I lived it. It makes me both frustrated and sad - because the hype is annoying, there is a lot of vaporware - but there is also something real that is happening too which is part of the story of the evolution of data architecture/infrastructure. My strong advice is, being open-minded is helpful - learn and take what was good/real and leave behind the stigma/hype. Something real and useful happened in terms of architecture, so take the benefits - but of course, don't compromise on delivering real, working solutions.

Regarding "data lakehouse", I struggle with the buzzword term also, but once again, I recommend looking at what is good/real and ignoring the stigma of the buzzwords. One way of looking at it is the literal translation - a data warehouse made from the components used to make data lakes. To be honest, it is a marketing term, but it is also an architectural pattern we had even before the term existed - for example - you could use a data warehouse product such as Vertica, and back it by HDFS - guess what, there's a data "lakehouse".. and most of the big database vendors can do this trick now - a full traditional data warehouse engine sitting on top of lake storage infrastructure.

There are several "real" use cases for data lakes. Precursor architectures could be seen to be "operational data stores" [1]. Data lakes are real, they are one approach to solving some problems.

These use cases could include: 1) raw, long term storage of large volumes of diversely structured data for staging/historical purposes; 2) data discovery/exploration of this data to identify patterns/models and relationships (this is both an AI and analytics use case for power users and BI/analytics/data scientists, etc.); and 3) an opportunity to change the paradigm of traditional ETL - instead of pulling from sources, one way you look at it is, allowing many diverse/distributed sources to push their data into the lake for powering analytics, exploration, AI model building, etc. It makes sense as part of lambda/kappa architecture as well - some of the "push" in can come from streaming sources as well.

Use case #1 is very much a "data infrastructure" kind of use case that we do anyway in data warehouses - especially those that do ELT (vs. ETL) - staging databases. If you want an architecture that actually helps make some sense of such a use case of data lakes more formally, one could look into Dan Lindstedt's data vault architecture [2]. While data vault modelling doesn't necessarily require "data lakes", the "raw" part of the data vault architecture use case overlaps nicely with data lakes.

[1] https://en.wikipedia.org/wiki/Operational_data_store

[2] https://en.wikipedia.org/wiki/Data_vault_modeling

spot on. Data lake approach is a lazy approach. Throw a pile of garbage to the storage layer and figure out how to use it later. People just kicking the can.

What you’re referring to is sometimes called a “data swamp”.

The simple way to understand the utility of a data lake is "S3 is phenomenally cheap".

Some people treat this as an excuse to throw whatever they want into it, without any organization or standardization... and the consequences of that are quite predictable.

But it doesn't have to be that way. You can accumulate diverse and large data sets, in cheap cloud storage, while knowing what everything is and where you can find it.

As a trivial example, let's say you have a typical OLTP database (or perhaps many), with useful data that is, unfortunately, mutable. You can store entire copies of those tables in your data lake for pennies a day, giving you the ability to recall a transactionally-consistent view of that data from various past times. This is something we've always been able to do using traditional tools, the difference is that storing the data in a "data lake" (i.e. S3) is orders-of-magnitude cheaper.

Another major use case, perhaps the most significant one, is storing the raw ingested data -- e.g. from telemetry collection, 3rd party exports, etc -- along with each stage of its transformation. By preserving the original input, along with all intermediate outputs, no information is ever lost. If a buggy transformation is discovered it no longer means your output is irrevocably corrupted, fixed results can be re-computed from wherever in the transform pipeline the bug manifested. And again, this was always possible, a data lake just makes it cheap enough to actually do on a large scale.

S3 storage is exactly what modern databases like Snowflake work on. It’s ridiculously easy to turn compressed unstructured (say, json) data into a SQL queryable table.

I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data scientists deal with noise more as a parameter of the models they are building. This difference leads to different requirements on how the pipelines should operate. In such a context, having a messy pile of data might help increase the velocity and the independence of some teams inside the company.

Noise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.

Totally agree on that, both have still to work with data quality but in different ways and with a different mindset.

Data is the new oil right so you want to decouple your data consumers from the data generators. This means you dump all the data into a lake. That way a reporting team or analytics team doesn’t need to connect to your production database.

In large organizations it is common to have multiple databases maintained by many different people. When a DS needs some data he/she needs to talk to all these different people, wait for them or their managers to grant access, and join the inconsistent labels. This takes a long time and some people are extremely protective of their (messy anyways) tables, requiring higher ups to straight up demand them to grant access.

Having nicely organized data is perfect, but I'd rather fetch the data myself from a pile of messy data instead of dealing with all these organizational nightmare.

Other resources on feature stores that help clarify the difference and when to choose it over a data warehouse

https://medium.com/data-for-ai - several blogs discussing feature stores for machine learning

https://www.logicalclocks.com/blog/feature-store-the-missing... - in depth explanation

featurestore.org - list of all feature stores available and in production

We have been building a feature first data store for seven years and it feels like feature store is about to become one of the more exciting ways to extract value from data. We see feature stores doing much more than becoming just another silo for ML, but instead a way to get a real-time, centralized view of fragmented data that has to either be federated or put in a data lake to to be queried together. I share more of my thoughts in this blog post: https://www.molecula.com/why-moleculas-feature-based-approac... Molecula is based on the OSS platform Pilosa (https://www.pilosa.com/) and both Pilosa and Molecula are transitioning to reposition as feature stores over the coming months. Doing machine-scale analytics and ML on the data itself will be a thing of the past.

But is it open-source? Our Hopsworks Feature Store is * https://github.com/logicalclocks/hopsworks as is GoJEK Feast * https://github.com/feast-dev/feast

I really believe that all Enterprise platform software should have an open-source version if it is to have a meaningful effect on how people work (in this case, how Data Scientists and Data Engineers work together).

Yes, Pilosa is OSS under Apache 2.

This looks like a highly specialised tool. How is it going to integrate with a Data Scientist's favourite tools, such as Jupyter notebooks, Pandas or Spark and especially ML frameworks like TensorFlow, SkLearn etc.?

Great question. Right now we have pushed hard to make SQL the primary interface to our feature store and can output Pandas and other formats at query time, however we are working on integrated/hosted Jupyter notebooks and excited to continue collaborating with the community on better feature first/oriented endpoints/interfaces. There is so much room to innovate here.

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select?

To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?

That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things.

Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.

I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.

I am not sure how this answer my question. I understand that a feature store contain the features that have signal.

My question is about the operational aspect of online vs offline stores.

I.e. what happen if the features needed for a prediction are not in the ONLINE store?

Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.

If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.

You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit:

https://databricks.com/session_eu20/real-time-feature-engine...

No, the feature would have to be in the online feature store (NDB, not Redis in our case). We have metadata about the schema of the model (the same schema as its training dataset) so that the applications can just supply the IDs and get back the feature vector in the correct order and get it also in the format needed (a npy array or a tfrecord example). This really helps maintenance, as adding a new feature means you don't have to update the application to account for the new feature and ensure the ordering of features matches the ordering in the training dataset.

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?

I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question.

Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.

it absolutely is.

This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations.

but modern data warehouses can support both.

https://www.snowflake.com/streaming-data/

I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions

Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload.

To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.

As a data scientist using snowflake and in the market for a feature store, the snowflake streaming is only for data ingestion, not serving. It doesn't solve the problem of serving data for a low latency app.

How is that not just a feature of some future Data Warehouse though?

hi, i'm the co-founder and CTO of a feature store startup that is building this on top of snowflake, can we chat? my name is Patrick, website is rasgoml.com, and e-mail is patrick@rasgoml.com. Thanks!

Hi. No, I don't think it is. Because your online applications that need low latency access to features won't tolerate the latencies provided by existing data warehouses. The online app that has an operation model that makes predictions is one client of the feature store. For the other client - a data scientist who is browsing features and creating train/test datasets - yes, that is similar to a data warehouse, except that you get APIs in Python and your data has precomputed statistics that make it better for exploratory data analysis than a traditional data warehouse.

Which existing data warehouses do you mean? ClickHouse and Druid can return answers in millisecond. Data warehouses are starting to optimize for low latency response.

Disclaimer: My company supports ClickHouse.

jamesblonde's article tried to answer the question:

> isn't this just a data warehouse [DW]?

My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning.

I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field.

Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries.

The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu.

[1] https://eng.uber.com/michelangelo-machine-learning-platform/

[2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef...

[3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...

Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions.

We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds.

You can find more info here on the research behind it:

https://www.logicalclocks.com/research

The architecture may be HTAP but the batch read use-case for the RowStore (vs. OLTP) seems to be unique to predictive machine learning (new to me at least). Full Text Search sync is a nice feature as well. Thanks for the thoughtful response and the research link.

I might quibble a bit about:

"They could derive historical insights into the business using BI tools."

A lot of work in some traditional BI tools (e.g. Oracle Hyperion) is actually about collecting and aggregating forward looking data (within forecasts, budgets, scenarios) that are then compared with actuals.

Yes, it was sloppy, sorry. "Derive insights from historical data" would have been more correct.

Great read! I can imagine that a Feature Store could have an impact regarding the challenge of provenance in ML as well. I couldn't find anything on that.

Appreciate the comparison table: Data Warehouse vs Feature Store. Might stea.. re-use that.

Nice write-up!

I would challenge the assumption that "the Data Warehouse is an input to the Feature Store" though. I'm more inclined towards having a first stage of data cleanup/modeling that could be reused (as input) for both DWH and FS instead.

Oh, this article is not about ML (the language) It is about Machine Learning... Can we just call Machine Learning Machine Learning, to avoid confusion?

Maybe you're being facetious. I had to look up what ML Language is, apparently it's a fifty year old programming language that hasn't had a stable release in 23 years, in case anyone else was wondering.

Respectfully, let's keep ML to be Machine Learning. ;)

Thanks for engaging instead of what seems now to be HN's standard response: downvote anything they don't know/understand... The standard has been stable for 23 years however various implementations are under active use and development (MLton's latest release was 10 days ago.). On top of that, a fair bit of founding research into Machine Learning was done in ML, adding to potential confusion.

Allas, I'll stop shouting "get off my lawn" and let the young ones make their own mistakes...

Reply on news.ycombinator.com