What Is a Feature Store?

With Tecton, transformations are an optional component of the system. Similar to Feast, you can bypass the Transform component to ingest data directly from external pipelines. You typically do this when you have existing data pipelines and you want to make the values available in a Feature Store. However, if you don't have an existing stream / batch data pipeline infrastructure that your data scientists / data engineers can easily contribute to, a Feature Store's Transform component is an easy way for them to be fully self-sufficient. Tecton makes it easy to express feature transformations using Spark's native DataFrame API, Python, SQL or Tecton's DSL.

Besides the self-sufficiency, there are a few other advantages you get from having a feature store manage your feature transformations:

- Feature Versioning: If you change a feature transformation, the Feature Store will know to increment the version of that feature and ensure that you don't accidentally mix features that were computed using two different implementations

- End-to-end lineage tracking and reproducibility: If a feature store manages your transformation, it can tie exact feature definitions all the way through a training data set and a model that's used in production. So, if years later you want to reproduce a model of a certain time in the past, a Feature Store that supports transformations would be able to recreate that model as long as the raw data still exists

- Trust: It's more likely that a data scientist will trust and then reuse another user's feature, if they can peek under the hood and see how the feature is actually calculated

- On-Demand Features: These transformations cannot be executed by existing data processing pipelines because they have to be computed in real-time when the prediction is made — which happens in the operational environment.

In reality, you will frequently see multi-stage data processing workflows in an organization: You will have a lot of data cleaning and preprocessing happening in an organization's standard and ML-independent data processing infrastructure. Afterwards, a Feature Store will pick up and transform that preprocessed data and turn it into feature values.