Hacker News Reader

Comment by mathisfun123 | Hacker News Reader

Gonna respond here and correct both comments

>Some context for those who aren't in the loop: ONNX Runtime (https://onnxruntime.ai/) is a standardization format for AI models.

It's just an IR, one of many - every framework has its own.

>Nowadays, it's extremely easy to export models in the ONNX format, especially language models with tools like Hugging Face transformers which have special workflows for it.

Meh it's poorly supported by both PyTorch and TF. Why support Microsoft's IR when you have your own.

>probably the most performant ML runtime at this point.

Not even by a long-shot - first party compilers are generally faster because of smoother interop but even amongst third-party you have TRT and TVM. TBH I have no idea what anyone uses ONNX for these days (legacy?).

View on HN

I have ask: where do you run your models, and how do you keep them... Deterministic? We have run models in a variety of environments from cloud to edge, and the only way we've found to do that is with ONNX.

Tensorflow has breaking changes in model behaviour on patch versions!

Pytorch and tensorflow require a huge python environment to run, and how on earth do we get that onto a client system without them having virtualisation?

Worst of all, they both have very significant changes in prediction depending on the CPU (or god forbid GPU) on the system.

Once we export to Onnx we find we get reliable output and performance across runtimes, which seems mandatory for running any kind of product.

Your experience matches mine exactly. Onnx seems to be relatively unknown for some reason. I feel like I'm on crazy pills -- how is everyone else delivering ML models? Are they really shipping multi GB pytorch environments? Are they using some sort of rube-goldberg machine to run things off a janky jupyter notebook?

In my experience, yes, all of the above. Azure ML Studio is a Jupyter Notebook Rube Goldberg machine builder. It's not cheap and it is not pretty, but it gets stuff done quickly. You are lucky if some developer took the time to properly nearly-productionize the Jupyter notebook as a Flask app or something (with yeah, a multi-gigabyte container). (Even luckier, in an Azure shop, if they skipped Flask and gave you a Functions app. Not quite fewer dependencies or smaller size, but Functions gives you free Application Insights telemetry and that is an operational beauty.)

Every time I've convinced data scientists to just hand off an ONNX file to me for Production everyone comes out pleased: building an ONNX file is easier than productionizing a notebook any other way; the speed and performance of ONNX runtime are great and it is more easily integrated in C#-based pipelines (potentially avoiding expensive data transfers to/from Python VMs).

The biggest, ugliest hurdle I've seen is how many pre/post-processing steps data scientists tend to accidental convince themselves "can only be done in Python" either because they don't have time to research alternatives, believe Python to be inherently magical, or found some massive, obscure Huggingface-like corpus with gigabytes of data that would result in the most bloated ONNX files and "obscured in a Python or VM install step" hides how big the corpus is.

> Meh it's poorly supported by both PyTorch and TF.

This does not match my experience. Most new model architectures port fine to ONNX from pytorch, only occasionally having to fill in rare functions.

> Not even by a long-shot - first party compilers are generally faster because of smoother interop but even amongst third-party you have TRT and TVM.

Again, in my experience, there is little by way of documentation for any of these platforms nor as broad support for as many OS and Hardware combinations.

> I have no idea what anyone uses ONNX for these days

Huggingface has strong support for ONNX and leverages it for improved performance in places.

https://huggingface.co/docs/optimum/onnxruntime/usage_guides...

It's curious how impressions can differ so.

It's not poorly supported by pytorch but you do have to have it in mind while writing your code. I've been having a hell of a time getting a decent ONNX model out of some pytorch code that was written with lots of dynamic logic inside the modules, it can be quite some work converting this to the necessary "static graph" design needed to export to ONNX properly.

What I'd love to see is a lean, inference-only version of pytorch that just works with existing code, and is smaller without the overhead of autograd and GPU support.

I was going to answer the same, I find the approach of machine learning compilers that directly compile models to host and device code better than having to bring a huge runtime. There are exciting projects in this area like TVM Unity [1], IREE [2], or torch.export [3]

[1] https://github.com/apache/tvm/tree/unity

[2] https://github.com/openxla/iree

[3] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

This is the precise pitch for Facebook's AITemplate as well.

Nope AITemplate is just a tracing mechanism plus cutlass plus autotuning. You still need a runtime. AIT also works on very few models (it's a replacement for something called "static runtime").

It's surprising to me how many people just shoot from the hip/guess on this stuff. IKYK and if you don't maybe don't guess?

No, AIT is a runtime - it exports a .so that you can load from C++ and call how you like.

The ONNX format have been develop by Microsoft and Faceboof and is really well supported by pytorch as it was develop for exchange between Pytorch and Caffe2. The ONNX Runtime is use by lots of software for inference only to run on windows without having to manage specific vendor provider. (fi, WinML runtime and ONNX runtime is the same code) TVM is also use but currently I have the impression it's more widely use in embed device.

ONNX Runtime enable also provider specific inference code like TensorRT, CUDA, CoreML etc... without having you to change your code

What do you mean by "first party compilers"? I've never heard this phrase in this context. Any examples?

Reply on news.ycombinator.com