Comment by neomantra | Hacker News Reader

Comment by neomantra | Hacker News Reader

Parent Full thread

I've had these strong feelings and the OP describes it really well. Despite being a polyglot programmer, I really struggle with Python, both in expression and performance (unless it's just config for GPUs).

Some of this frustration was recently an "Unpopular Opinion" on the Go Time Podcast regarding Python being great for "data exploration" but not for "data engineering": https://changelog.com/gotime/304#t=3196

I've been yearning for better interactive tooling and ML-related libraries to bridge this gap and started using some even in just the last week:

* GoNB (Golang-support for Jupyter notebooks, also from a Googler) https://github.com/janpfeifer/gonb

* That uses Go-Plotly for graphs/UI: https://github.com/MetalBlueberry/go-plotly

* GoMLX (GoNB author is also on that project, many thanks Jan!) https://github.com/gomlx/gomlx

* Hidden at the end of OP is LangChainGo for LLMs, which I haven't used yet: https://github.com/tmc/langchaingo

Pick those up and let's make the Go community stronger together!

My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis.

Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.

Is that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.

When I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯

Keep on educating them I'd say ;)

Go will sadly never be viable for ML tasks due to its C interop. Maybe calling an LLM api but often you have a mix of models. Rust also struggles here a bit due to its poor C++ interop.

Written but someone who has built ML libraries from scratch in Go

I'm starting to explore this for Time-Series analysis -- still at the beginning as I mostly used C++ before. Torch.ch (so old now!) used LuaJIT which has excellent FFI.

What is the realm of the interop problem there? That one wants to assemble data in Go and then feed it to another API? Or even at a lower level, getting data to a GPU/TPU requires C interop so it's more about the memory sharing?

Those libraries come from Google, so hopefully they can help battle these issues in the nether realms.

Can you elaborate on this or point me to some more information? I am primarily a Go dev that has been integrating AI/ML into a personal project. Like many other apps, I wrap llama.cpp and various other popular ML projects.

Recently I have been getting the itch to go lower level and see if it would be feasible to write an LLM backend in pure Go, or a Stable Diffusion backend. Just for curiosity and learning.

But if this is a dead-end I would like to know from someone more experienced than I. Thanks for any advice you can offer in advance.

Do you know of any resources with details on using llama.cpp with Go?

Why not elixir then? Good c interop, functional, and good performance.

Erlang/Elixir are very slow for most computationally expensive tasks.

Isn’t python the same? Most of the work is done by c++ or rust libs with python as glue.

Python is the same but there's not nearly enough incentive to move to another slow language. A lot of people that are interested in using something other than Python want a language that doesn't require everything to be written in C, C++ or Fortran and used via bindings. Having most of the ecosystem written in one expressive and relatively fast language has huge benefits.

At my last company I built a Go tensorflow inference server.. it worked well

I would be really surprised if Go becomes the next ML modelling language. Python has always been a stretch. My money is on Julia in the next 5 years.

I also noticed at the last Gophercon in san diego that a lot of talks were regarding data engineering and go. There was even a university professor with a data eng courses that presented about how he teaches most of it in Go and believes its better. (obv since hes at gophercon :) )

Reply on news.ycombinator.com