Hacker News Reader

Comment by astronautas | Hacker News Reader

Sure, but what about non AI / business logic pre and post? Think RAG calls, guardrails, ...? Or do they fly compared to LLM inference itself?

View on HN

Probably depends on the specific case. Basic initialization is stupid fast compared to any stupid training exercise. If your training is going to take anything longer than a minute you are probably very very safe keeping init/postprocess in python and well if you're doing anything intense consider importing a faster way of doing it. Python is like a motherboard for soldering components.

agree, it depends, always benchmark, but my question is rather generic i.e. I am looking for a perspective.

Ok my perspective of pursuing something like this in Rust and continuing the motherboard analogy, is that you take on the full burden of developing everything in your laptop onto a single chip (sounds pretty awesome really) but your boundaries can end up blurred as everything is just a single etch. Forcing boundaries by making components with hard won decisions creates a marketplace of exchangeable components. Force everyone to agree on that API by dropping across a hard language boundary.

Your impact is measured by raw performance masked by time in market. Pure Rust or whatever will have higher raw performance but lower overall (my guess) impact because it misses out on a wider market. Python API will have in general a very slightly less raw performance but much wider time in market.

can't disagree (a tradeoff).

It's getting very philosophical, without a (perhaps impossibly complex/fraught) model of real-world programmers using all available languages doing just what you ask there is no way to find the sweet-spot. I've given some rules of thumb why python still rules and why things in general favor standard forms easily reviewable by humans and peppered with magic only understandable by geeks.

edit: you've removed a line while I was resonding like: "maybe go is a sweet-spot"

I think if you are looking to replace python as a runtime you'd maybe be better off arguing for safety, python is much more easily corruptible so if maybe you don't trust your machine(s) you are training on and don't want to be mislead by someone hacking your initializations then arguably running a compiled program with certain guarantees is safer.

I agree Python is hardly replacable. Btw, I only mean inference, not training. Training imho should stay pure Python, you can achieve mega throughout with it for batch processing.

> Btw, I only mean inference

D'oh! Yea that changes things. I would be considering UI integration from the inference side.

When you say RAG calls are you talking about requests to the (usually vector db) external datastore, or the repeated calls to the LLM?

“Guardrails” are often just calls to one or more (usually smaller) classification/moderation models.

call to external db, and then to llm with retrieved context.

also business rules, no?

> call to external db, and then to llm with retrieved context.

Right, neither of those are CPU intensive in a different way than LLM inference itself (the latter is LLM inference itself.)

> also business rules, no?

Business rules can vary quite a bit in content and complexity, but either tend to be simple enough that they won’t impose much additional load, or complex enough that you are probably going to want to simply use an existing rules engine (many of which, regardless of their implementation language, have Python bindings) which are going to behave the same way no matter what language you call them from.

Fair points!

Reply on news.ycombinator.com