C++ Implementation of StableDiffusion | Hacker News Reader

C++ Implementation of StableDiffusion

github.com

157 pointssmusamashah36 comments

Someone please edit the title as it's really misleading. This linked project is using onnxruntime as opposed to implementing stable diffusion from scratch in C++. Meaning this is not in the same spirit as llama.cpp.

onnxruntime is written in C++. So it is not misleading =)

Also, doesn't llama.cpp use ggml.cpp ?

Or at least has parts of that project copy pasted into its tree?

Yes, so llama.cpp is also not in the same spirit as the llama.cpp version op has in his head.

Yes, both were created by the same person. On llama.cpp README says

> This project is for educational purposes and serves as the main playground for developing new features for the ggml library.

(Edit, typo.)

A bit misleading, it's relying on onnxruntime for the actual inference. That said, given the state of python packaging and the fact that onnxruntime has APIs in C++ and Java, I don't know why more people don't distribute self-contained examples using it.

What are the implications of onnxruntime vs something like llama.cpp or those nn-512 models that seem closer to just C code for inference? What's different?

onnxruntime is typically for GPU acceleration while being nearly unusable on CPUs. It’s also better supported (Microsoft) and supports LOTS of APIs.

Llama.cpp / ggml (while they support some hardware acceleration) is more focused on commodity hardware like x86 CPUs and Apple M-series silicon.

Hardware acceleration includes: OpenBLAS/Apple BLAS/ARM Performance Lib/ATLAS/BLIS/Intel MKL/NVHPC/ACML/SCSL/SGIMATH and more in BLAS.

Also new apple metal implementation in progress in addition to apple accelerate, and is in baseline if you enable it.

Also a CUDA implementation.

Any POSIX system that has a C++11 compiler could compile llama.cpp and run it, including all the BSDs. The OS supported by onnxruntime is very limited.

It looks like onnxruntime supports all major OSes and the web.

Yes, many are finally getting the point that Python "libraries" are actually bindings to native libraries, and any language can have bindings to the same libraries.

I no longer feel like a crazy person for implementing other ML stuff in C some years ago.

It's really nice to see that C++ implementations in the ML field are growing. Personally I do not like Python very much. C++ does not force you to do things in a specific (Pythonic) way.

Growing? You're always free to skip Python and reach out for the actual library instead of Python bindings.

And since C++17, it is quite easy to write Python like code.

Seems to be a Microsoft Windows specific implementation.

Was hoping for something cross platform on to Linux.

As mentioned in the README it's using onnxruntime for inferencing. Onnxruntime is a available on all platforms including Linux. See https://onnxruntime.ai/.

> The current codebase and the resulting Nuget packages target Windows and use DirectML, however only small sections of the code utilize Windows specific APIs, and thus could be ported to other platforms with minimal effort.

Okay... what's stopping them from doing so? The effort?

If it were trivial, that statement wouldn't exist.

Well, that's a little uncharitable, isn't it? Assuming the developers primarily use Windows, they probably just don't have much priority for it; there's probably a lot of things they could do with the time. Perhaps they don't even have a setup to test it, and even as a staunch Linux user, I can't blame someone for not wanting to mess with all of the GPGPU drivers and non-sense.

Just trying to be fair here.

> I can't blame someone for not wanting to mess with all of the GPGPU drivers and non-sense.

Thanks for the downvote and for proving my point for me. That was literally exactly what I said. :)

But yeah, if you say something is "simple" and you can't do it yourself, then it's most likely not.

The misunderstanding here is that I am referring to getting Linux GPGPU to work in the first place, not porting the code itself. Presumably they are suggesting that one of us Linux weirdos who already have gotten ROCm and/or CUDA working could do it with relative ease. That said, I have not attempted to verify the veracity of that assertion, but I think it's what they meant.

Though even if they could just do it, and have everything they need to do it, that doesn't mean they're lying. It really could just not be a priority.

So it's back to the effort right? It's not easy afterall!

Well, with that line of thinking, definitely. Because of course, FIRST, you have to invent the universe.

There is sometimes a wide gulf between simple to implement and easy to verify. They may not have a Linux box set up with the hardware, drivers and development environment to build ML code. In that case even if the fix is simple, testing it and verifying it works is less simple. But anyone interested in using it on Linux would need to do that setup as a prerequisite anyway.

Plenty of open source projects only release on one OS because that's the one the developer uses, even if porting them would not be difficult.

Ah, so completely unusable. Got it.

Assuming SD models are converted to the onnx format (or whatever its called) shouldn't current sd projects offer similar performance? The dominant cost is in generation is inference, not the project runtime I thought.

I think the purpose isn't to improve performance but to make it not depend on a bunch of Python packages, which can be difficult to install.

This. Conda and Venv in paper should be like docker plug and play... But in practice they can be a hit or miss. Still better than manual global library management.

Conda requires a commercial license too

And is a 900MB download on Widndows.

Apparently binaries take space, what a surprise. /s

Fwiw it's not impossible to package all this junk up in Docker. I did it recently. You have to run Docker w/ a whole bunch of flags though, like `--gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -p 7860:7860`. And the GPU isn't available during build time unless you do some hacks. But you can build the CUDA junk into the docker image along with Python and all the libs. Combine that with a .bat or .sh to contain the giant `docker run` command and you have a half-OK solution. Aside from having to start docker before being able to run it.

That's a nice tip. I'll try it. I had faced issues trying to run Automatic 1111 + oogabooga webui because of conda conflicts.

For testing purposes of nice self contained packages like this, are pre-trained datasets available?

Reply on news.ycombinator.com