Hacker News Reader

Comment by Roark66 | Hacker News Reader

Onnx is cool. For one it runs (large) transformer models in the cpu twice faster than pytorch/transformers. But at the moment it lacks a number of crucial features. Specifically:

It's reliance on google's protobuf with it's 2gb single file limit is an extreme limitation. Yes you can keep weights outside your model file, but still many operations (model slicing) fail.

Second, inability to offload parts of the model to disc or cpu(like huggingface accelerate) while the rest executes on the gpu.

Thirdly, inability to partition existing large models easily. You can delete nodes, but then fixing the input/output formats means manually editing text files. The work flow is ridiculous (convert onnx to txt with pdoc, edit in text editor, convert back to binary).

I really wish they fix all this stuff and more.

View on HN

As ONNX models are protobufs you can edit them at a Python or Java REPL (or other language but I've personally used those two). Dumping them out as text seems like a lot more work, and a lot less typesafe.

protobuf has a serialized file limit of 2gb? or like a definition file? I haven't touched PB in ages and certainly not something that would be transmitting that much in a single message.

is your complaint that the whole context/weights needs to be sent through the whole file?

I'm asking from a position of ignorance, I'm surpised to see a serialized transport of that size, and wondering why it's specifically limited at 2gb. like to be able to mmap on 32-bit hardware?

>protobuf has a serialized file limit of 2gb

I'm not sure if this is a file size limit too or just an object memory representation size limit. For me using a library designed for message passing to save/read your AI models is a bad design decision.

>is your complaint that the whole context/weights needs to be sent through the whole file?

I store large onnx models with "external" weights, but even so many operations fail with the dreaded "ModelProto exceeds maximum protobuf size of 2GB: 3385275542". So the complaint is that you simply can't do a lot of stuff with models over 2gb.

You can create a session to execute the model, you can run the vanilla optimisation over it. But trying to run transformer specific optimisation errors out as well as making any attempts at slicing the model. Additionaly some conversion processes.

It appears the code that does that looks up the model size and just errors out if over 2GB.it doesn't even try loading it

> I'm not sure if this is a file size limit too or just an object memory representation size limit.

Message. Each message cannot be bigger than 2GB(and usually should be not this big), but a file could contain multiple messages. This limitation helps them prevent integer overflows, since every length is 32-bit but every application process is 64-bit, you can convert the length numbers to 64-bit before doing any arithmetic operation. Therefore, fundamentally, there is no way to make it secure on 32-bit platforms, or no way to support more than 2GB on 64-bit platforms without totally rewriting the code.

Reply on news.ycombinator.com