Hacker News Reader

Comment by diggan | Hacker News Reader

> And now everyone is rewriting everything in Go/Rust left and right.

Especially interesting for software that are 99.9% of the time waiting for inference to come back to you. Sure, makes sense to rewrite something that is heavily relying on the CPU, or where you want an easier time to deal with concurrency, but I feel like that's not what makes Codex takes long time to finish a prompt.

With that said, I also rewrote my local LLM agent software to Rust, as it's easier to deal with concurrency compared to my initial Python prototype. That it compiles into a neat binary is an additional benefit, but could have as easily gone with Go instead of Rust.

View on HN

> Especially interesting for software that are 99.9% of the time waiting for inference to come back to you.

In a different domain, I’ve seen a cli tool that requests an oauth token in Python be rewritten to rust and have a huge performance boost. The rust version had requested a token and presented it back to the app in a few milliseconds, but it took Python about five seconds just to load the modules the oauth vendor recommends.

That’s a huge performance boost, never mind how much simpler it is to distribute a compiled binary.

Python's startup cost is terrible. Same with Node. Go is very good, but Rust is excellent.

Even if a GC'ed language like Go is very fast at allocating/deallocating memory, Rust has no need to allocate/deallocate some amount of memory in the first place. The programmer gives the compiler the tools to optimize memory management, and machines are better at optimizing memory than humans. (Some kinds of optimizations anyway.)

TBH I'm still surprised how quickly Go programs start up given how much stuff is there in init() functions even in the standard library (e.g. unicode tables, etc)

I’ve spent some time optimizing Python performance in a web app and CLI, and yeah it absolutely sucks.

Module import cost is enormous, and while you can do lots of cute tricks to defer it from startup time in a long-running app because Python is highly dynamic, for one-time CLI operations that don’t run a daemon or something there’s just nothing you can do.

I really enjoy Python as a language and an ecosystem, and feel it very much has its place…which is absolutely not anywhere that performance matters.

EDIT: and there’s a viable alternative. Python is the ML language.

Packaging Python apps is pure hell. npm gets a lot of shit, but Python deserves as much if not more.

I know far too much about python packaging while only knowing a little about it.

I agree it’s hell. But I’ve not found many comprehensive packaging solutions that aren’t gnarly in some way.

IMHO the Python Packaging community have done an excellent job of producing tools to make packaging easy for folks, especially if you’re using GitHub actions. Check out: https://github.com/pypa/cibuildwheel

Pypa have an extensive list of GitHub actions for various use cases.

I think most of us end up in the “pure hell” because we read the docs on how to build a package instead of using the tools the experts created to hide the chaos. A bit like building a deb by hand is a lot harder than using the tools which do it for you.

That’s fair. I’m also thinking about the sheer size of Python apps that make use of the GPU. I have to imagine a C++ app performing neural network shenanigans would’t be >1GB before downloading weights.

I’ve tried, it is still gigabytes, unless you try to dynamically link to user installed CUDA libraries from c++. Which I don’t recommend.

Oof

Speaking of which, I didn't realise Node had a built in packaging feature to turn scripts into a single executable:

https://nodejs.org/api/single-executable-applications.html

I was not aware of that feature either, thanks for the heads up

In my opinion, bundling the application Payload would be sufficient for interpreted languages like python and JavaScript

Reply on news.ycombinator.com