Hacker News Reader

Comment by hashtag-til | Hacker News Reader

This is very cool! I really hope the ONNX project gets much more adoption in the next months and years and help reduce the fragmentation in the ML ecosystem.

View on HN

Eh... I have seen ONNX demos for years, and they tend to stay barebones and slow, kinda like this.

NCNN, MLIR and TVM based ports have been far more impressive.

Have you used TVM much? Is it only good for inference?

Inference only, as far as I know.

TBH the mlc projects were my first exposure, but they are all impressive, as are the bits of source I poked through. And the autotuning makes a huge difference in my quick testing.

Lololol show me an "MLIR" port. Do you mean tensorflow port or jax port or torch port (that uses torch-mlir)? Or do you really mean llama implemented in linalg/tosa/tendor?

I wasn't talking about Llama specifically. I was thinking of the SHARK Stable Diffusion port (which uses MLIR/IREE), as it considerably outpaced the ONNX runtime.

But apparently the performance of llama on torch-mlir is progressing: https://github.com/nod-ai/SHARK/issues/1707

I'm not sure there's much chance of that happening. ONNX seems to be the broadest in coverage, but for basically any model ONNX supports, there's a faster alternative.

For the latest generative/transformer stuff (whisper, llama, etc) it's often specialized C(++) stuff, but torch 2.0 compilation keeps geting better, BetterTransformers, TensorRT, etc.

Reply on news.ycombinator.com