- Years and years ago, we already had a data warehouse (DWH)
- In the data warehouse, you would store data like, each and every order that all customers have made (i.e. up to and including full-fledged facts and dimensions about each)
- Now, let's say axiomatically/hypothetically, a very useful and highly predictive feature for ML is "# of orders made in past 7 days" for each customer
- Can this be computed from the data already in the DWH? Yes, absolutely, but it's a new computation and not an existing column/attribute in the dimensional model.
- What if you need to recompute this feature daily, for millions of customers and orders? Well, we could always just add it to the dimensional model, compute it once, and let people just use it/share it... but why? Most internal users of the DWH probably don't care about something like "# of orders past 7 days" as something to be added to a customer dimension or per-customer grain (too specific or whatever), and moreover, the DS/ML folks want the same feature but for "every 1/3/7/30/90/180/365/730/etc." days breakdown, as well as a bunch of variations about orders and things other than orders (e.g. "average time between new orders, over past 7/30/90 days" or "average $ spent over past 7/30/90 days", as features that serves as a proxy for frequency of activity and level of engagement)
- Hence, it makes sense to keep the "golden copy" of data in a canonical form in the classic/standard DWH as a baseline, and to separately/independently compute features out of that data and to store them in a different system (which can also be optimized for the different query/access patterns that DS/ML have, vs traditional BI). Over time, it also made sense in certain cases to go upstream of the DWH to source data from and process it more directly (for performance/efficiency reasons), though generally deriving features out of the transformed dimensional models was still very useful.
- It took our teams ~1-2 years to really go through this evolution and reach a mature-ish state, but for the past 2-3 years, we've benefited tremendously from having an independent feature repository/store, that is separate from the classic DWH. Benefits came in all the obvious and some non-obvious ways, i.e. in faster iteration/cycle time, in better quality/repeatability, and in being able to automatically discover interesting relationships that no human could have anticipated - simply by having a very broad/large repository of features and running automatic feature selection over it.