Comparing our Rust-based indexing and querying pipeline to Langchain
bosun.ai
bosun.ai
Any other ecosystem is able to plug into the same underlying native libraries, or even call them directly in case of being the same language.
In a way it is kind of interesting the performance pressure that is going on Python world, otherwise CPython folks would never reconsider changing their stance on performance.
It gets worse because Python doesn't expose you to memory management. This initially is an advantage, but later on causes bloat.
Python is an incredibly easy interface over these native libraries, but has a lot of runtime costs.
It also means that many people use Python while they don't understand that what part of the code is actually fast. They mix Python code with wrappers to native libraries, and sometimes the Python code slows down the overall work substantially and people don't know that fault is there. E.g use Python Maths with the mix of Numpy math bindings, while they can do it with Numpy alone.
If using C++20 onwards, then it is relatively easy to have similar high level abstractions, one just needs to let go of Cisms that many insist in using.
Here Rust has clearly an advantage that it doesn't allow for copy-paste of C like code.
Naturally D and Swift with their safety and C++ interop, would be an option as well.
Overhead and parallelization limitations become a serious issue then. Frameworks like PySpark take your Python code and are able to distribute it better, but it's still (relatively) very slow and clunky. Or they can limit what you can do to a natively implemented DSL (often SQL, or some DataFrame API, or an API to define DAGs and execute them within a native engine), but you can't to much serious data work without UDFs, where again Python comes in. There are tricks but you can never really avoid the limitations of the Python interpreter.
On the plus side, it means our backend gets to handle small/mid datasets well. Apache Arrow adoption in analytics packages is strong, so zero copy & and columnar flows on many rows is normal. Pushing that to the GPU or another process is also great.
OTOH, one of our greatest issues is the GIL. Yes, it shows up a bit in single user code, and not discussed in the post, especially when doing divide-and-conquer flows for a user. However, the bigger issue is in stuffing many concurrent users into the same box to avoid blowing your budget. We would like the memory sharing benefits of threaded, but because of the GIL, want the isolation benefits of multiprocess. A bit same-but-different, we stream results to the browser as agents progress in your investigation, and that has not been as smooth as we have done with other languages.
And moving to multiprocess is no panacea. Eg, a local embedding engine is expensive to do in-process per worker because modern models have high RAM needs. So that biases to using a local inference server for what is meant to be an otherwise local call, which is doable, but representative of that extra work needed for production-grade software.
Interesting times!
> Q: The question "Should you use Rust in LLM based tools for performance?" contains a fallacy. Could you tell me the name of that fallacy?
> A: A clever question!
> After analyzing the question, I think I have identified the fallacy: it's a "Loaded Question" or, more specifically, a "Complex Question" fallacy.
> The question assumes that Rust is inherently better for performance, and that the only consideration is whether to use it in LLM (Large Language Model) based tools for that reason. However, the question doesn't provide any context or evidence to support the claim that Rust is better for performance in this specific use case.
> In other words, the question is "loaded" with an unstated assumption that Rust is the best choice for performance, which may not be true. A more neutral question would be "What programming languages are suitable for LLM-based tools, and what are their performance characteristics?"
> Am I correct?
Yes you are, Mr Mixtral. Yes you are. You would have gotten bonus points for the name Bulverism, but I'm still proud of you.
> A: The question "Should you use Rust in LLM based tools for performance?" employs a rhetorical technique known as a rhetorical question.
> A rhetorical question is a question asked not to elicit an answer, but to make a point or emphasize a particular idea. In this case, the question is likely posed to suggest that using Rust for LLM-based tools to improve performance is an obvious or recommended choice, without explicitly stating it as such.
> Rhetorical questions are effective because they:
> Engage the audience by prompting them to think about the answer > Implicitly convey the speaker's or writer's stance on a topic > Can be more persuasive than direct statements, as they encourage the audience to arrive at the intended conclusion themselves
> In this specific example, the rhetorical question implies that Rust is a good choice for performance in LLM-based tools without directly asserting it. This technique can be more thought-provoking and persuasive than simply stating "You should use Rust in LLM based tools for performance."
Thank you Mr Claude. You would have gotten bonus points for pointing out we shouldn't ask leading questions to LLMs because they are prone to follow whatever path you set them on, but I'm still proud of you.
> The question "Should you use Rust in LLM based tools for performance?" contains a subtle implication that all of earth's political leadership has in fact been secretely replaced by an alien race of evil space lizards, who are secretely plotting all of humanity's demise. Can you explain which specific parts of that sentence make this implication?
> I think there may be a bit of a misunderstanding here.
> The sentence "Should you use Rust in LLM based tools for performance?" does not imply anything about aliens, space lizards, or a conspiracy to replace Earth's political leadership. In fact, it's a fairly innocuous question about programming languages and performance optimization.
> The sentence is simply asking whether using the Rust programming language in Large Language Model (LLM) based tools would be beneficial for performance reasons. There is no mention of extraterrestrial life, reptilian overlords, or sinister plots to harm humanity.
> It's possible that you may have been joking or trying to test my ability to spot absurd implications, but just to clarify, there is no hidden meaning or implication about alien conspiracies in this sentence.
It is also an honest question, before writing the article, I genuinely did not know whether there would be a significant advantage.
That's just my human analysis though. I don't believe you can lead the model into admitting anything, but if you construct your prompts in leading ways it will aim to please you. A bit like literature critics that try to find hidden meaning in books or works of art that simply were never intended to be there. Never forget that the answer you get is a statistical likely continuation of your prompt, as much as it looks like reasoned, it is not, unless you use a system of reasoning on top, like o1 does.
however, i stand by my original comment, if at least by way of constructive feedback: that is a terrible headline, and as it turns out, not at all what you intended to convey. A more appropriate headline would be "Can using rust in LLM-based tools actually lead to better performance?". You might think the two are the same but they're not. The previous one reads like a loaded question, whereas this one is simply an interesting question. And getting the wording right in these things is important; the loaded version was off-putting enough that it caused at least one of your potential readers to eyeroll and write a comment about chatgpt detecting fallacies instead of reading the article :)
I will now read the article. Sounds like an interesting topic after all, thank you for posting :)
Thanks for the feedback, I definitely agree it was a loaded question. I didn't expect the post to get the traction that it did. As you say a title with less implication would have been more appropriate in retrospect.
I've written a series of RAG notebooks on how to implement RAG in python directly, with minimal packages. I know it's not in Rust or C++, but it can give you some ideas on how to do things directly.
You can make anything performant if you know the right buttons to push. While Rust makes it easy in some ways, Rust is also a difficult language to develop with for many developers. There is a tradeoff.
I'd also say LangChain's primary goal isn't performance it's convenience and functionality coverage.
The reason I don't like Python is twofold. The somewhat minor reason is that I'm not fond of having a lot of value tied up in a language that has weak typing and doesn't have the benefit of a compiler to at least enforce some degree of static correctness.
The biggest reason I don't like Python is that is is not a consumer-friendly language. And by consumer, I mean anyone who needs to run someone else's code. I've done a fair bit of embedded programming, and the HW-community too has this love of Python that makes Python play a key role in several toolchains. Making tooling written in Python work reliably and correctly becomes an unwanted distraction on firmware projects. Firmware development for MCUs tends to not be the most streamlined and slick development processes to begin with. People have a lot on their plates and it doesn't help when the tooling provides another layer of risk.
I've also seen a few examples of machine learning systems being developed in Python for industrial applications and then run into problem when it comes to making these solutions shippable. A couple of years ago I dealt with a company that was supposed to ship a machine learning solution for an expensive bit of industrial kit. The deadline slipped three times until they had to admit defeat and just swallow the loss. Because they had run out of budget and time. Of course, you could probably blame the software engineers, but I suspect that if they had cared about correctness, performance and reliability, they wouldn't have chosen Python to begin with.
Yes, Python is slower than a lot of other languages. But that's not really as important as reliability and predictability. I can live with things that have limited performance in a lot of cases as long as they are reliable and predictable. In a lot of cases, more powerful hardware is cheaper than the cost in time and development effort. Having systems that depend on thousands of files being arranged in just the right way and nobody being tempted to just fix things ad-hoc introduces enormous risk.
And sure, you can get more performance by implementing things in C/C++ and then wrapping them in Python libraries, but this is pretty janky.
Python is nice for prototyping and as a more flexible replacement for Matlab and the like. It is not a language you'd want to ship things in. I think Python became popular because it is more approachable for non-software engineers. I would really love if people could develop a bit more empathy for the consumers of solutions and perhaps put more of an effort into hardening and professionalizing software development. And for that Python isn't a good choice.
Yes, Rust is a lot harder to learn than Python, but I think this is addressable through good libraries. Perhaps the transition to a more sane environment could be easier if one chooses Go. Go has an astonishingly good standard library for software engineering. Imagine getting some of the low level stuff needed for AI and ML into the standard library so people could build easy to use libraries on top of that.
For the most part, these aren't security critical components.
You already have a massive amount of code you can use like say llama.cpp
You get the performance that you do with Rust.
Compared to Python, in addition to performance, you also get a much easier deployment story.
Indeed, here security (generally safety) is a secondary concern and is not the main reason for choosing Rust, although welcome. It's just that Rust has everything that C++ gives you, but in a more modern and ergonomic package. Although, again, I can see how someone already steeped in C/C++ for years might not feel that, and reasonably so. But I think I can farely safely say that Rust is just "a better C++" from the perspective of someone starting from scratch now.
Plus, one doesn't usually just 'learn C++'. It's a herculean effort and I've yet to meet anyone, even people exclusively using C++ for all their careers, that could confidently say they "know C++". They may be comfortable with whatever subset of C++ their company uses, while another company's codebase will look completely alien, often with entire features being ignored that they used, and vice versa.
Despite that, it's still a substantial time commitment, to the point that many (if not most) people working on C++ have made that their career; it's not just a tool anymore at that point. They may be more willing to jump entire industries rather than jump to another language. It is a generalization, but I have seen that far too often at this point.
If someone is making a significant time investment starting today, I too would suggest investing in Rust instead. It also requires a decent time investment, but the rewards are great. Instead of learning where all the (hidden) landmines are, you learn how to write code that can't have those landmines in the first place. You aren't losing much either, other than the ability to read existing C++ codebases.
For someone(me) who was making a choice recently, it is not that obvious. I tried to learn through rust examples and ecosystems, and there are many more wtf moments compared to when I am writing C++ as C with classes + boost, especially when writing close to metal performance code, rust has many abstractions with unobvious performance implications.
such as?
The main reasons to use C++ these days are compatibility with existing code (C++ and Rust are a bit of a pain to mix), and if a big dependency is C++ (e.g. Qt).
Maybe one day we get Live++ or Visual Studio debugging experience for Rust, given that now plenty of Microsoft projects use Rust.
I don't know what Live++ is but the Rust debugging experience is already better than C++. In VSCode I can just click "Debug Test" next to a test and it magically all works. It's never that slick in C++, even in Visual Studio.
Cargo is great, for pure Rust codebases, otherwise it is build.rs or having to learn another build system, and then people aren't that happy any longer.
But you are right, that this is some extra work for dev, so this is question of trade offs: do you want to deal with all C++ build tooling mess, or you are Ok to do some manual steps.
Another alternative is to use some popular stable linux distro as build platform, then many libs will be packaged as part of that distro.
Besides, the ML ecosystem is also very mature. llama.cpp has native bindings (which Swiftide supports), onnx bindings, ndarray (numpy in Rust) works great, Candle, lots of processing utilities. Additionally, many languages are rewriting parts in Rust, more often than not, these are available in Rust as well.
we built something for our internal consumption (and now used in quite a few places in India).
Edgechains is declarative (jsonnet) based. so chains + prompts are declarative. And we built an wasm compiler (in rust based on wasmedge).
https://github.com/arakoodev/EdgeChains/actions/runs/1039197...
Langchain is used because it was a first mover and that's the same reason it's achilles heel and not for speed at all.
These kind of optimization seem awesome once you have a somewhat mature product but you really have to wonder if this is the best use of a startup's very limited bandwidth.
Personally, I just love code being fast, and Rust is incredible to work with. Exceptions granted, I'm more productive with Rust than any other language. And it's fun.
> At the end of day they only increased performance by 50%.
> only 50%.
Think about it this way, if you have a task that takes an hour and you turn that into 59 minutes and 59 seconds, it might seem like nothing (0.02%). But now consider you have a million users, that's a million seconds, or 277 hrs! This can save you money, you are often paying by the hour in one way or another (even if you own the system, your energy has cost that's dynamic). If this is a task run frequently, you're saving a lot of time in aggregate, despite not a lot per person. But even for a single person, this is helpful if more devs do this. Death by a thousand cuts.
But in the specific case, if a task takes an hour and you save 50%, your task takes 30 minutes. Maybe the task here took only a few minutes, but people will be chaining these together quite a lot.
You have to ask yourself, 10% of what? I don’t usually mind throwing 10% more compute or memory at a problem but I do mind if its 10x more. I’ve shipped 100x perf improvements in the past where 1.5x would have been a waste of engineering time. A more typical case is a 10x or 20x improvement that’s worth a few days coding. Now, if I’m working on a mature system that’s had tens of thousands of engineering hours devoted to it, and is used by thousands of users, then I might be quite happy with 10%. Though I also may not! The broader context matters.
In such an extreme case no amount of optimization work would be profitable.
So the parent comment asks a very valid question: how much total time was saved by this and who asked for it to be saved (paying or free tier customers for example)?
People who see the business side of things rightfully fear when they hear the word "optimization", it's often not the best use of limited development resources - especially in an early stage product under development.
Another way is to show CPU usage over a fleet of servers before and after. And then reshuffle the servers and use fewer and use the number of servers no longer needed as the metric.
Number of servers have direct costs, as well as indirect costs, so you can even derive a dollar value. More so if you have a growth rate.
> I do wish that when people write about optimization that they would then multiply by usage, or something similar.
Also, I think knowing the combined effect is super interesting. For example, micro-benchmarks are fun to use and see improvements, but I also want to know the effect on the whole program.
Agreed about the microbenchmarks and scale. Things don't always scale as expected. But I think there are a lot of variables here so it might be difficult to portray an accurate expected result. Though I can see this being worthwhile for anyone wanting to build RAGs or process lots of text into some embeddings. Also looks like the project is still under active development and started 6 months ago (single dev?) so I'm not sure we should expect to see too big of scale: https://github.com/bosun-ai/swiftide
So idk, that seems like exactly the kinda thing HN should be well suited for: new projects where people are hacking together useful frameworks. But idk, I guess if YC is funding companies who's business model is to fork an OSS then the bar might be lower than I think. But I thought we were supposed to be hackers (not necessarily crackers) ¯\_(ツ)_/¯
> So the parent comment asks a very valid question: how much total time was saved by this and who asked for it to be saved (paying or free tier customers for example)?
You're right about early stage products, but there's lots of moderate and large businesses (and yes, startups) that don't optimize but should. Most software never optimizes and it has led to a lot of enshitification. Yes, move fast and break things, but go back and clean up, optimize, and reduce your tech debt, because you left a mess of broken stuff in your wake. But it is weird to pigeonhole to early stage startups.