Hacker News Reader

Comment by imjonse | Hacker News Reader

> And this is a superficial difference carried from old days when we need to do deployment and deployment-specific optimizations.

Is it? From what I understand, to use an analogy, ONNX is the bytecode specification and JVM whereas Pytorch, TF and other frameworks combined with converting tools are the Java compilers.

View on HN

Onnx is just a serialisation format (using protobuf iirc) for the network, weights, etc.

Your training framework and a suitable export is the compiler.

Onnx Runtime (which really has various backends), tensorrt, .. (whatever inference engine you are using) is your JVM.

That is my understanding, ONNX is the weights and the operators. You could then project that model into SPIR-V, Verilog or run it via native code.

It is true if your model is statically shaped. ONNX is also a collection of C++ code , a C++ library for doing shape inference. You cannot project the C++ code to another high level programming language.

The model itself doesn't contain C++ code. The runtime supports Python, JS, etc. I don't think they are shipping a copy of clang as part of the rt. I could be wrong, I haven't looked. Of course it needs the scaffolding to get data in and out.

I just did an install of the runtime on Python ( pip install onnxruntime ) . Here are the additional packages it installs.

    Package       Version
    ------------- -------
    coloredlogs   15.0.1
    flatbuffers   23.5.26
    humanfriendly 10.0
    mpmath        1.3.0
    numpy         1.25.2
    onnxruntime   1.15.1
    packaging     23.1
    protobuf      4.23.4
    sympy         1.12

https://onnxruntime.ai/docs/install/

ONNX is a spec. ONNX Runtime is an implementation of ONNX. There are other implementations too. But ONNX is not a text spec like the RFCs for network protocols. ONNX is also a collection of C/C++ code. ONNX's implementations rely on this code to do type and shape inference. My point was: if someone wants to implement ONNX(write a library that can load and run ONNX models), he/she has to reuse this C/C++ code, or totally rewrite them in his/her favorite programming language(but I think it is not very practical).

If an ONNX implementation wants to do codegen, like what XLA does, then usually it is based on LLVM and it needs to be shipped with a copy of LLVM.

Reply on news.ycombinator.com