Go, Python, Rust, and production AI applications | Hacker News Reader

Go, Python, Rust, and production AI applications

ajmani.net

62 pointstodsacerdoti29 comments

I switched from python to rust for my AI stuff. Honestly, I don't care about the things people say rust is used for. I like it because the package manager, testing, and typings being built into the ecosystem by default makes it so easy to build. VS Python where it all can be done, but you need to then maintain all of those separate tools. The overhead of writing Rust is less than the overhead of dealing with the Python ecosystem. And then you have all the benefits of Rust everyone mentions more often... one other thing no one mentions is the feedback loop between a strongly typed language and copilots ability to more accurately generate code.

That being said, there is a real shortage of Rust software for Rust only projects. I ended up writing a wrapper for Llama.cpp and open ai API [0] because I needed it and couldn't find anything out there. Eventually, I do intend to implement Hugging Face's Candle library [1] (A rust version of Torch). There is something appealing about doing everything in a single lang especially as the monopoly of CUDA inevitably gets chipped away.

[0] https://github.com/ShelbyJenkins/llm_client [1] https://github.com/huggingface/candle

> no one mentions is the feedback loop between a strongly typed language and copilots ability to more accurately generate code.

Dang, I never thought of this.

I am the same, and for me the biggest reason to use rust for this stuff is simply how if i leave projects, I know that the type system is so strict that I know in a few months things should still work even as I just jump in and change stuff. With python projects they just feel like they rot.

Also as copilot generates code for me, if it compiles in rust I know it will usually work well. With python the iteration loop is so much less trustworthy

I've had these strong feelings and the OP describes it really well. Despite being a polyglot programmer, I really struggle with Python, both in expression and performance (unless it's just config for GPUs).

Some of this frustration was recently an "Unpopular Opinion" on the Go Time Podcast regarding Python being great for "data exploration" but not for "data engineering": https://changelog.com/gotime/304#t=3196

I've been yearning for better interactive tooling and ML-related libraries to bridge this gap and started using some even in just the last week:

* GoNB (Golang-support for Jupyter notebooks, also from a Googler) https://github.com/janpfeifer/gonb

* That uses Go-Plotly for graphs/UI: https://github.com/MetalBlueberry/go-plotly

* GoMLX (GoNB author is also on that project, many thanks Jan!) https://github.com/gomlx/gomlx

* Hidden at the end of OP is LangChainGo for LLMs, which I haven't used yet: https://github.com/tmc/langchaingo

Pick those up and let's make the Go community stronger together!

Go will sadly never be viable for ML tasks due to its C interop. Maybe calling an LLM api but often you have a mix of models. Rust also struggles here a bit due to its poor C++ interop.

Written but someone who has built ML libraries from scratch in Go

Can you elaborate on this or point me to some more information? I am primarily a Go dev that has been integrating AI/ML into a personal project. Like many other apps, I wrap llama.cpp and various other popular ML projects.

Recently I have been getting the itch to go lower level and see if it would be feasible to write an LLM backend in pure Go, or a Stable Diffusion backend. Just for curiosity and learning.

But if this is a dead-end I would like to know from someone more experienced than I. Thanks for any advice you can offer in advance.

Do you know of any resources with details on using llama.cpp with Go?

I'm starting to explore this for Time-Series analysis -- still at the beginning as I mostly used C++ before. Torch.ch (so old now!) used LuaJIT which has excellent FFI.

What is the realm of the interop problem there? That one wants to assemble data in Go and then feed it to another API? Or even at a lower level, getting data to a GPU/TPU requires C interop so it's more about the memory sharing?

Those libraries come from Google, so hopefully they can help battle these issues in the nether realms.

Why not elixir then? Good c interop, functional, and good performance.

Erlang/Elixir are very slow for most computationally expensive tasks.

Isn’t python the same? Most of the work is done by c++ or rust libs with python as glue.

Python is the same but there's not nearly enough incentive to move to another slow language. A lot of people that are interested in using something other than Python want a language that doesn't require everything to be written in C, C++ or Fortran and used via bindings. Having most of the ecosystem written in one expressive and relatively fast language has huge benefits.

At my last company I built a Go tensorflow inference server.. it worked well

My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis.

Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.

Is that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.

When I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯

Keep on educating them I'd say ;)

I also noticed at the last Gophercon in san diego that a lot of talks were regarding data engineering and go. There was even a university professor with a data eng courses that presented about how he teaches most of it in Go and believes its better. (obv since hes at gophercon :) )

I would be really surprised if Go becomes the next ML modelling language. Python has always been a stretch. My money is on Julia in the next 5 years.

I have spent the majority of my life programming in Python, starting from the very first program I wrote. However, recently I've begun learning Go, and in many respects, it feels more Zen-like compared to Python. In Python, it seems there are a million ways to accomplish a single task, whereas Go often presents a singular, correct approach. As projects grow in size, the expressiveness of Python and the weak type safety eventually become a liability.

When I first programmed in Python over 15 years ago, Python very much made me feel like there was one clear way to do a particular thing. Maybe as Go gets older, it too will have multiple ways to do the same thing.

I like Go and use it a lot but for AI, Rust seems to have better cards in conjunction with Python. Go’s runtime makes it very impractical to build bindings for Python (it’s e.g. not possible to load two Golang-based libraries simultaneously as the runtimes will overwrite each others memory, or at least that was the case a while ago), Rust is much better suited to that task. And I think by now Rust already has a head start for data science & AI tooling.

Nim has also a powerful deep learning library called Arraymancer. It's selling point is that you don't have to rewrite your code from research to production. It's used in various machine learning projects, but one recent one that caught my eye was https://github.com/amkrajewski/nimCSO "Composition Space Optimization"

https://github.com/mratsim/Arraymancer

A missing tech here feels like TypeScript. It feels like the perfect mix of expressiveness and type safety for AI/ML work. I wish there was more tooling out there for it.

For me, Go feels too verbose, but I appreciate the static typing and performance. Python with its robust ecosystem and optional types feels okay, but the performance is really awful.

What about Python performance is awful in AI? All of the hard stuff is delegated to a C library or the GPU.

There's tons of glue code that gets written in Python.

Python makes it hard to get this right. Simple idioms result in copies of large collections or other work shifting from C to Python. The state-of-the-art of Python profiling [1] has to target this problem to the exclusion of other concerns.

[1]: https://www.thestrangeloop.com/2022/python-performance-matte...

I think there are 2 layers in an AI application that can be considered somewhat orthogonal:

1. Making your models network accessible in a configured way 2. RAG and any chaining code for model calls

For #1, this is probably going to start with pytorch since that's typically what research code uses, and get hand optimized over time as your application matures.

For #2, it seems like the most popular tools so far are in the Python space, of course that doesn't mean other languages don't have robust ecosystems for AI already and of course new things get developed all the time, but ecosystem size is important to consider.

Honestly I think Elixir has made some great strides in the past few years to also be an interesting alternative. Bumblebee, Livebook, the NX project are also awesome pieces of tech

Reply on news.ycombinator.com