Hacker News Reader

Comment by jamesblonde | Hacker News Reader

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

View on HN

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?

I might be falling for something related to Cunningham's Law here but.. I believe the whole article is an effort of trying to answer this question.

Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.

it absolutely is.

This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations.

but modern data warehouses can support both.

https://www.snowflake.com/streaming-data/

I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions

Online data is not necessarily real-time streaming. I am making a distinction between OLTP workloads for the online applications that need a feature vector (i.e., a row of data) to make an individual prediction, and a client that is creating train/test data from millions of rows of data (features) - that is the OLAP workload.

To be more concrete, Feast is an open-source Feature Store built on BigQuery and originally BigTable. But the latency of BigTable for the OLTP workload was too high for GoJEK (feature lookup is just one part of making a prediction), so they switched to Redis. Redis PK lookups are a couple of ms, on average, compared with 10+ ms for BigTable. What is the latency of a PK lookup on snowflake? It ain't a millisecond or two. On MySQL Cluster (NDB), our online feature store, PK lookups return in sub-ms latency on dedicated hardware.

As a data scientist using snowflake and in the market for a feature store, the snowflake streaming is only for data ingestion, not serving. It doesn't solve the problem of serving data for a low latency app.

How is that not just a feature of some future Data Warehouse though?

hi, i'm the co-founder and CTO of a feature store startup that is building this on top of snowflake, can we chat? my name is Patrick, website is rasgoml.com, and e-mail is patrick@rasgoml.com. Thanks!

Hi. No, I don't think it is. Because your online applications that need low latency access to features won't tolerate the latencies provided by existing data warehouses. The online app that has an operation model that makes predictions is one client of the feature store. For the other client - a data scientist who is browsing features and creating train/test datasets - yes, that is similar to a data warehouse, except that you get APIs in Python and your data has precomputed statistics that make it better for exploratory data analysis than a traditional data warehouse.

Which existing data warehouses do you mean? ClickHouse and Druid can return answers in millisecond. Data warehouses are starting to optimize for low latency response.

Disclaimer: My company supports ClickHouse.

jamesblonde's article tried to answer the question:

> isn't this just a data warehouse [DW]?

My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning.

I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field.

Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries.

The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu.

[1] https://eng.uber.com/michelangelo-machine-learning-platform/

[2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef...

[3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...

Yes, sradman. You describe it correctly here as a HTAP architecture. What we do differently is that the same scaleout consistent metadata layer is used for (1) online FS (NDB - mysql clustrer), (2) offline FS (hive), and (3) HopsFS-S3. That is, the data files for Hive are stored in HopsFS, that in turn stores its data in S3. Because of our metadata architecture, we ensure the consistency of hive data files with hive metadata using foreign keys, and we have extended metadata to store all the feature metadata that is kept consistent with good auld foreign keys and transactions.

We also automatically synchronize the metadata to elasticsearch, so you can do free-text search for features, files, extended metadata, etc. So, you can search up to PBs of files/dirs/tables/features/extended-metadata in milliseconds.

You can find more info here on the research behind it:

https://www.logicalclocks.com/research

The architecture may be HTAP but the batch read use-case for the RowStore (vs. OLTP) seems to be unique to predictive machine learning (new to me at least). Full Text Search sync is a nice feature as well. Thanks for the thoughtful response and the research link.

I might quibble a bit about:

"They could derive historical insights into the business using BI tools."

A lot of work in some traditional BI tools (e.g. Oracle Hyperion) is actually about collecting and aggregating forward looking data (within forecasts, budgets, scenarios) that are then compared with actuals.

Yes, it was sloppy, sorry. "Derive insights from historical data" would have been more correct.

Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select?

To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?

That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things.

Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.

I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.

I am not sure how this answer my question. I understand that a feature store contain the features that have signal.

My question is about the operational aspect of online vs offline stores.

I.e. what happen if the features needed for a prediction are not in the ONLINE store?

Yeah in this type of architecture there is some sort of join required. Interestingly, @maycotte talking about the Molecula feature store – it serves real-time and historical features in the same storage layer. Significant performance gains with that approach.

If you want to reuse features, and not write a new pipeline for every model, you need JOINs. Both for the online and offline feature stores.

You can do real-time feature engineering in your application as well. But we recommend to do it in a feature pipeline that also syncs the features to the feature store. We have a talk about it coming out at the spark/ai summit:

https://databricks.com/session_eu20/real-time-feature-engine...

No, the feature would have to be in the online feature store (NDB, not Redis in our case). We have metadata about the schema of the model (the same schema as its training dataset) so that the applications can just supply the IDs and get back the feature vector in the correct order and get it also in the format needed (a npy array or a tfrecord example). This really helps maintenance, as adding a new feature means you don't have to update the application to account for the new feature and ensure the ordering of features matches the ordering in the training dataset.

Reply on news.ycombinator.com