Hacker News Reader

Comment by minimaxir | Hacker News Reader

It's somewhat disappointing that research is the primary motivator for the switch. PyTorch still has a ways to go in tooling for toy usage of models and deployment of models to production compared to TensorFlow (incidentally, GPT-2, the most public of OpenAI's released models, uses TensorFlow 1.X as a base). For AI newbies, I've seen people recommend PyTorch over TensorFlow just because "all the big players are using it," without listing the caveats.

The future of AI research will likely be interoperability between multiple frameworks to support both needs (e.g. HuggingFace Transformers which started as PyTorch-only but now also supports TF 2.X with relative feature parity).

View on HN

OpenAI is a fundamentally a research laboratory - their top priority is pushing the envelope on machine learning research and releasing their progress responsibly. Anything that reduces experimentation and research speed should be avoided, thus the switch to PyTorch makes sense.

As long as as OpenAI is open sourcing their work, there will always be others that will port it over to other frameworks.

> It's somewhat disappointing that research is the primary motivator for the switch.

They are a research organization, how is that disappointing?

Making AI more open is synonymous with making AI more accessible, which (IMO) is much better facilitated with TensorFlow/Keras versus PyTorch.

Many AI tutorials imply that the more complicated an AI approach is, the more effective it is, which isn't practical, especially for newbies without a deep background.

Accessible to whom? What makes tensorflow/keras more accessible than pytorch?

Accessible to non-researchers, especially those with a programming background but not an AI background.

The TF/Keras approach advocates the minimum amount of code necessary and effort needed to make model changes, with sensible default configurations and layer architectures.

I disagree with you because we switched from TF 1/Keras to PyTorch and our codebase reduced to half of its size. Our team is a bunch of developers with little AI background. Problem with TensorFlow is it is mostly not readable and onboarding new people to project is really hard. In contrast, PyTorch much more readable and people with python background can easily adapt to a pytorch project after a small machine learning lesson.

I am a non-researcher with a programming background working in ML for the past ~2 years, pytorch was a godsend for me and felt much more programmatic and pythonic than TensorFlow. Keras is also good, but claiming that PyTorch makes it harder for non-researchers is wrong IMO.

STRONGLY disagree. I’m just a hobbyist, but trying to read Keras models can be a god damn nightmare if the author has to do anything even slightly non-standard. Keras seems to REALLY want you to believe that you can just throw a bunch of layers together and call .fit and everything will just work, but it never seems to be that simple unless you’re training on MNIST or ImageNet.

Check out the fastai library. It's something like Keras is for Tensorflow.

As a non-researcher, mostly programmer who has spent a lot of time delving into this ecosystem, PyTorch is the most like "standard programming". With fastai giving you models to do working three liners.

I haven't used tensorflow interactive execution though, it supposedly is closer to PyTorch than the graph building model.

Minimum=/=lowest effort.

Especially with the caveat of "with a programming background", it is far easier to reason and debug through PyTorch with just Python knowledge, compared to TensorFlow/Keras, which sooner or later requires you to learn a condensed history of TensorFlow/Keras development to understand why things are the way they are.

In my opinion,

  import lib
  lib.train("imagenet", "resnet50", epochs=10)
  lib.eval()

is NOT a good example of a beginner friendly library. It's a thin wrapper facade that hides all of the actual complexity behind "Train ImageNet in 3 lines of code!"

Fair; maybe minimum isn't the right word. More like "minimum without full abstraction."

The Keras examples are a good reference (e.g. https://www.tensorflow.org/tutorials/keras/classification ); even without an AI background, you have a sense of both what's going on how to tweak the model to improve it.

The reason why Keras became so popular is that it borrowed a lot of concepts from Lua Torch (which predates even Theano). And anyone who worked with Torch immediately sees it reading Keras code. But Torch was Lua and naturally it received less recognition than it deserved. Your will not lose anything by simply moving to PyTorch.

The idea that Keras would be the framework of choice for a research organization is laughable. If you're talking TF internals, those aren't really any more "accessible" than pytorch, which to many (including me) feels quite idiomatic

OpenAI doesn't have any products right? So research is a lot more important than production for them. Surprises me it took them this long.

The aforementioned GPT-2 and Spinning Up were released, open-sourced tools using TF; however, other recent models such as the OpenAI Five Dota 2 bot, the robot dexterity demo, and the hide-and-seek simulation were not open-sourced.

If OpenAI is signaling that they are changing their open-source strategy, they should be more explicit about that.

Reply on news.ycombinator.com