Hacker News Reader

Comment by doomslice | Hacker News Reader

That's just the SDK that they're saying is merging (with TFX being the one that "wins"). If you've used them you'll know they have huge overlap but TFX abstracts a ton of what it's actually doing.

View on HN

TFX is just a SDK (with a few libraries - like data validation, model analysis, statistics). So, i don't know about saying it wins? The battle of ML pipeline ecosystems is the engine - not so much the API. It had been Beam vs Spark. Now, Google are changing tack and saying it is TensorFlow on Kubernetes with distributed processing vs Spark-based ML pipelines (that may use PyTorch/Tf for training).

I work on TFX. TFX started as an internal Google project [1] but over the years we've been steadily open sourcing more and more of it. Portability and interoperability are key focus areas for our project, more information in [2]. TFX OSS currently consists of libraries, components, and some binaries. It is up to the developer to pick the right level of abstraction for the task at hand. More information about this in [3]. Many of the ideas in TFX were inspired by an earlier production ML system at Google called Sibyl, more information about Sibyl in [4]. The two systems combined represent over a decade of production ML experience of many engineers and we hope other engineers will find it useful, too.

Also of interest may be this overview blog post [5] by yours truly :)

[1] ACM KDD '17 "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform" https://dl.acm.org/doi/10.1145/3097983.3098021 [2] https://www.tensorflow.org/tfx/guide#portability_and_interop... [3] https://www.youtube.com/watch?v=zxd3Q2gdArY [4] https://www.youtube.com/watch?v=3SaZ5UAQrQM [5] https://blog.tensorflow.org/2019/05/research-to-production-w...

I am getting downvoted here without anybody actually addressing my arguments - TFX/Beam being replaced by TFX/KubeflowPipelines and the new platform being plugged in tightly to Google's managed services (AI platform, BigQuery, GCS, etc). I didn't think my tone was negative - they are observations that nobody in this subthread is addressing. As a systems person, TFX to me was a layer on top of Beam to tie ML pipelines together. Now, Beam is gone - dataflow appears as a managed service you can use in your pipelines (see the figure on the blog post). TFX is being repackaged as the API for Kubeflow pipelines. That was my take and what i wanted to discuss.

For what it's worth, I can't even downvote :)

I don't know the _eventual_ direction of Kubeflow/TFX -- but in our TFX pipelines you still get to choose where it runs. From the docs:

> Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. TFX uses Apache Beam to implement data-parallel pipelines. The pipeline is then executed by one of Beam's supported distributed processing back-ends, which include Apache Flink, Apache Spark, Google Cloud Dataflow, and others.

You can also choose to run it on Kubeflow itself.

Reply on news.ycombinator.com