Building the Future of TensorFlow
blog.tensorflow.org
blog.tensorflow.org
I hope torch has a convincing distributed tensor API coming soon. Their development on ShardedTensor seems to have slowed or stopped recently, so TF’s DTensor is definitely ahead, which is a shame. And of course TF’s ecosystem for the whole lifecycle is more mature with TFX, TF.js, etc, but torch is slowly closing those gaps, and hopefully that will continue.
Though I agree that it is very weird that google is treating TF as an internal product, versus something more akin to GCP. There's no reason for them to do so especially after they had the chance to break and redo tons of stuff for TF2.
I also think Keras' Functional API is superior in terms of composability than PyTorch's OOP model, but I am biased as a software engineer. It does feel like the community thinks the OOP model is much more hackable thus easier to use.
All in all, it is still early days. We don't have a competent all-in-one OSS SQL database until late 2000s, which is 20-ish years after the theory was ready and taught extensively in the school. And even after that, we have plenty of innovations around database in 2010s for new use cases. Frameworks for differentiable programming have long way to go.
What makes you say that?
I’ll note that checking right now there was a commit 15 hours ago, but the last commit before that seems to be 28 days. So some work is still going on at least, thankfully :)
Lots of development and traffic happening here: https://github.com/pytorch/tau/
I actually think that torch’s ShardedTensor looks very promising. Essentially you can initialize a sharded tensor from an already initialized tensor, or initialize a sharded tensor on a meta device where it’s not allocated locally and each shard gets initialized on the specified remote devices (useful for extremely large tensors)
The sharding is described by a ShardingSpec, where you can either let it shard equally sized shards across the requested devices, where the split happens along a single dimension, or you can do grid sharding along multiple dimensions. They also have a more general sharding spec that allows you to choose explicitly which indices go on which devices, if you need non uniform shards.
I think once these are implemented (along with some special cases like cloned tensors, and things like that), and once the distributed autograd engine has full support for CUDA, it should be pretty easy to start building out distributed versions of common neural net operations.
The one thing (that I haven’t thought about a ton, to be frank, and I’m sure other smarter people have :)) is that you’ll end up in cases with both a sharding spec for the weights as well as for the inputs, and what’s the best way to make sure everything matches up. Is the best way to handle that custom logic for each operation? And do you have each operation just reshard the input automatically? Seems potentially like a pretty big performance pitfall.
Let's hope JAX won't suffer the same fate.
I object to this statement. The earlier releases of TF (ca. 2013) were impossible to debug and the documentation was always broken - if you tried to follow the Seq2Seq tutorial you know what I'm talking about. I'd argue that these releases were great for the 50 people who already knew how to use it, but they were aggressively unhelpful for beginners.
PyTorch won my lab (and certainly others) because you could add prints to check your dimensions while TF forced you to build a correct computation graph all at once. Performance? Sure, TF is probably faster. But I'd argue that TF's big mistake was not taking their new users into consideration.
At least with jax the core library isn’t adopting any of the framework level stuff so those can evolve independently.
Agree on the framework stuff. Please just be a library, not a collection of opinionated frameworks where I need to read the source code anyway to understand what it actually does. After something not working and debugging for hours I remember looking at the number of weights in the model and thinking, wait, something can't be right here. Then I dig into the framework layers and figured out it added slightly different things than I thought it would. Would've been much faster to just write the graph myself.
(source: https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-202...)
Also the threat of Pytorch can be seen when reading between the lines, especially since it's now run by a foundation and the darling of the diffusion model developments.
It can easily be wildly non deterministic across different cpus or GPUs, or even in the same session, with the same input.
Performance seems to get worse with new releases, and there are frequent subtle breaking changes when using models built with old versions on newer releases.
Tensorflow serving is barely controllable, and requires insane tuning to make it perform the same as pytorch, but provides little to no docs.
The vast majority of models people build just don't work in tensorflow serving either, as you can't reach in with hacky python to mess with internal state.
If you use a custom host instead then you have to deal with literal gigabytes of python dependencies, making your docker images huge.
Memory usage is uncontrollable and causes terrible performance or instant host death. Results vary depending on cpu count, and automatic parallelism can reduce performance.
I just don't understand how Google use tensorflow internally for real world services.
What are you using for serving PyTorch models?
Not because it’s google but because Jax has so much momentum lately.
https://www.deepmind.com/blog/using-jax-to-accelerate-our-re...
TF will continue to have a place at Google for prod work but its application base is going to continue to shrink. I'm just blown away they're rejiggering the distributed model again.
(all that, while depending on a single-file, 3-release python library...)
I love to hear that. But does Python even guarantee 100% backwards-compatible?
For domain-specific languages like TF I guess they were motivated to commit to it to ensure adoption of the new versions.
C++ isn't stagnant, if anything it should slow down a bit as it moves faster than an overcaffeinated hamster on a rocket.