For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.
For anyone interested in research in deep learning with a python-numpy like environment, I can only recommend to switch.
We used to be a TensorFlow shop. Last year we started playing around with Pytorch, only for R&D projects at first... and it felt like a breath of fresh air. Whereas TensorFlow always felt like it was designed and engineered from day one for massive scalability and deployment flexibility (at the expense of easy/fast development), Pytorch felt instantly like it was designed for easy/fast development by AI researchers who need to experiment and iterate through models as quickly as possible with as little hassle as possible. Almost overnight, we stopped tinkering and experimenting with TensorFlow.
Earlier this year, after a blog post announcing the production-friendly features of Pytorch 1.0[a], we decided to switch our production systems from TensorFlow to Pytorch. So far we're happy with the decision.
I can do everything in both pytorch and tensorflow and when I have to define really efficient input pipelines (tf.data is a great thing), parallelize and distribute the training and export a trained model to production... With tensorflow everything is easier.
Moreover, pytorch in 1.x will have static-graph too, exactly like Tensorflow.
Both frameworks are converging to something really similar. I don't see a reason to switch (right now)
For point 2: I don't see the problem. I use almost daily a wrapper (defined inside tf.contrib, that in version 2.x will go in core [I hope]) around the optimizer that in 2 lines allows me to distribute the training on multiple GPUs on the same machine
https://www.logicalclocks.com/goodbye-horovod-hello-tensorfl...
What actual product(ive) things?
To mitigate this complexity, PyTorch and Tensorflow are heavy weight machine learning frameworks that give your software a lot of structure as well as tooling to monitor the progress of training and debug your models as well as some deployment tooling.
Any neural net based component of software will likely be developed in one of these frameworks.
Or generative models: same workflow (python definition and train. Export and use in other languages).
I use Tensorflow basically everyday. I use pytorch only when I read a model implemented in this framework to reimplement it in Tensorflow (in order to use all the tools I developed to simplify the train to shipping to production phases)
I'm constantly surprised by how much attention it gets given how narrow the scope is. I guess a lot of people need to classify images.
> My theory is that TF is used mainly because it is supported by Google, even if it is really badly designed for practitioners.
This is not correct. TF is mainly used because it's designed for industrial-strength machine learning. It provides primitives with an eye to scale from the outset (because it's Google).
It's probably true that prototyping / research was not the main audience. That's exactly why Keras was adopted, as well as features like tf.eager, to abstract away the underlying computation graphs and make it easy for people to try different things.
Well-designed primitives / abstractions are important; Tensorflow does this well.
Not sure if he develops and maintains during company time, but wouldnt be surprised
The reason I bet on Tensorflow originally I admit was it was built by Google. But another really strong reason is it had a fully integrated toolchain from research to production. That toolchain has gotten much stronger with the continued investment in tf.data and tensorflow-serving. There was a binary alternative to CSVs (TFRecords). Also given some of our needs with some pretty esoteric components the tf community is actually quite deep.
I had only minor python experience before starting with tensorflow so I literally did not want a python-numpy environment. I am still sad that the world has converged on python as the language of choice but that ship seems to have sailed.
I will say tf.while is something I won't miss, as is tf.cond. I didn't find either really difficult, but it was more typing than was necessary. If there is something I really think should be invested in it is documentation. What should be in a SessionRunHook and what is the best way to write them? If you dig through some of google's repos that use TF estimators you find a lot of useful hooks as well as utility functions to help build them. But learning how to use them was more painful than it should have been.
All that said I am not a researcher. I would suspect that some of my workflow overlaps with researchers as I have found myself implementing a number of papers. I also need the ability to rapidly move a model from training to production.
Sounds like every Google developed technology. Angular, Go, Android, the list is infinite.
I use TensorFlow via Keras because there are a lot of concise examples, Q&As and tutorials on it for different use cases while Pytorch looks a little bit like a magic black box you can't use before you take a course and study it as a whole. I believe this is a major point for many. Also the first Pytorch [official] tutorial I've found mentioned an nVidia GPU as a prerequisite and I don't have a GPU, just old built-in Intel Graphics with Core 2 Duo and TensorFlow seems having no problems with this.
Sonnet is based on TF