Hacker News Reader

Comment by 0xbear | Hacker News Reader

In my experience (computer vision, deep learning) PyTorch is substantially faster as well, especially in data augmentation where it’s not just a thin layer over cudnn.

That said, you’re right. There’s no way I’d deploy it to production.

View on HN

Have you looked at ONNX? It is a neural network exchange format that, in particular, lets you deploy PyTorch models in production via Caffe2. Here is a tutorial: http://pytorch.org/docs/master/onnx.html

Disclaimer: I work on Caffe2 team (not on ONNX, though)

If I want to export my own computational graph to ONNX, what is the first place I should look at? Do you know about any documentation or reference implementation of the format?

Reference specification and validator are hosted in https://github.com/onnx/onnx

We are taking a pretty hard look at ONNX right now. It doesn't support a lot of ops yet. I would add that it's very early for supporting anything "real" yet.

Does the ONNX thing actually work?

I'm super skeptical of these interchanges, because it seems very difficult to avoid train/test skew. Any difference in detail between the two implementations is a potential problem. I can imagine different order of operations putting out some values by 0.1%, causing 1% eventual loss of accuracy.

It should, if you find that converted model doesn't work, it is a bug, please file it with PyTorch and/or Caffe2.

Implementation details of course differ between frameworks, but luckily neural networks are very robust to noise. In my experience, changes like using Winograd/Fourier for convolutions, or even running the whole thing in FP16 do not result in noticeable artifacts, and these are among the biggest differences you could have between frameworks.

> There’s no way I’d deploy it to production.

I've seen this stated by a few people, but never justified. What are some reasons for not deploying PyTorch in production?

Nonsense, Pytorch is great in production. I use it for Mathpix (mathpix.com) which processes 20 million images per month.

Don't take this the wrong way, but 20M images per month is only about 20 qps or so (20M / 3600 / 730 => 7.6 qps, but that's unrealistically even). That's pretty manageable even on a single box, depending on the network and the size of the box. When people say they're concerned about a system in production, they often mean when needing to roll it out to many machines due to higher scalability requirements.

Disclosure: I work at Google, but not TensorFlow.

That's a fair point, but just as a counterpoint, for plenty of people, 20M per month is a lot of traffic, and it's good and valid to hear about experiences of people putting PyTorch into production with this scale.

"Production" tends to be an overloaded term. Arguments like this tend to lead to "My production is bigger than your production!" which is frankly silly.

You could mean large scale, real time, "small batch job with online lookups from a CRUD database",..

Let's just admit it's use case specific and move on.

Just one instance of our product has to process 39 million inferences per month (number is dictated by frame rate), with predictable latency, 24x7, on both Windows and Linux.

About 8 images per second? How does that prove that Pytorch is great in production?

I am not saying that Pytorch is bad in production but I fail to see how your metric of 8 images per second proves anything.

While you're absolutely right that vanity metrics prove nothing, it does prove that you can choose Pytorch for a production environment with a decent amount of traffic and be happy with your choice.

Why wouldn't you deploy it in production - on server side at least?

Because TF lets you run _just_ the model, with no Python stuff at all. For us there is no “server side”, our stuff is deployed on prem. The fewer variables there are the easier our lives are going to be.

Additionally, PyTorch download page warns you point blank that it’s an early version of the software and that you should “expect some adventures”. Adventures are fine for research, but inadvisable in production IMO.

I understand the appeal, but IMO wrapping the Python stuff in a docker container is fairly simple.

Making it run at scale using something like kubernetes is more advanced stuff but still within good devops practices.

Unless, of course if you want to run on exotic HW, like TPUs. But that's an issue for Googlish scales.

Edit: BTW, if on premise you mean Windows client software your point is totally valid, Python would suck for this.

Depending on how much extra logic you add around the model serving library, then you are exposing your source code, too. Maybe you care, maybe not.

Re: Windows, there's no TF serving for that, so TF is no better than other Python frameworks.

Reply on news.ycombinator.com