Re-implementing LangChain in 100 lines of code

I work with Langchain on a daily basis now, and so often I find myself asking; do I really need a whole LLM framework for this? At this point, the assistant I am writing, will likely be more stable rewritten in pure Python. The deeper and more complex the application becomes, the more of a risk Langchain seems to become to keeping it maintainable. But even at less complex levels, if I want to do this:

1. Have a huge dataset of documents.

2. Want to ask questions and have an LLM chat conversation based on these documents.

3. Be able to implement tools like math, wiki or Google search on top of the retrieval.

4. Implement memory management for longer conversations.

Its still a lot more straightforward to maintain it in Python. The only thing where it becomes interesting is having agents execute async, which is not that easy replicate, but at the moment agents are not that helpful. Not trying to diss Langchain too much here, because its such an awesome framework, but I can't help seeing past it other than just being a helpful tool to understand LLM's and LLM programming for now.

LangChain has been so frequently discussed that I thought it must be this amazing piece of software. I was recently reading about vector databases and how they can be used to provide context to LLMs. I came across a LangChain class called RetrievalQA, which takes in a vector database and a question and produces and answer based on documents stored in the vector db. My curiosity was piqued! How did it work? Well... it works like this:

    prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
    {context}
    Question: {question}
    Helpful Answer:"""

My sense of wonder was instantly deflated. "Helpful Answer:". Seriously? I think LLMs are cool, but this made me realize people are just throwing darts in the dark here.

Am I the only one who is not convinced by the value proposition of langchain? 99% of it are interface definitions and implementations for external tools, most of which are super straightforward. I can write integrations for what my app needs in less than an hour myself, why bring in a heavily opinionated external framework? It kind of feels like the npm "left-pad" to me. Everyone just uses it because it seems popular, not because they need it.

Yeah, the basics of LangChain are fairly simple, and reimplementing a loop like that in Go, including tool usage, was very straightforward when I was writing Cuttlefish[0] (a toy desktop chat app for ChatGPT that can use stuff like your local terminal or Google).

The magic in LangChain, though, is the ecosystem. I.e. they have integrations with tons of indexes, they have many tool implementations, etc. This is the real value of LangChain. The core ReAct loop is quite trivial (as this article demonstrates).

[0]: https://github.com/cube2222/cuttlefish

I got the chance to try Langchain as part of a hiring process. I was already having my eye on it for a personal projects though.

The moment I tried it and went through the docs, the entire abstraction feels weird for me. I know a bit here and there about LLM, but Langchain make me feels like Im learning something entirely new.

How agent and tools work and how to write one wasnt straightforward from the docs, and the idea of having an AI attach itself to an eval or writing its own error/hallucination-prone API request based on a docs doesnt give me a lot of confidence.

The hiring assignment specifically mentioned to use Langchain thought, so I did. But just as a glorified abstraction to call GPT and parses the NL output as JSON.

I did the actual API call, post-processing, etc. manually. Which I have granular control over it. Also cheaper in terms of token usages. You could say I ended writing my own agent/tool that doesnt exactly match Langchain specifications but it works.

I guess Langchain had its use case. But it feels pretty weird to use for me.

I've been working with langchain and llamaindex and did notice that it's a pretty hefty abstraction on top of pretty simple concepts and I also eventually ended up dropping both and simply write the underlying code without the framework on top.

There’s always DSP for those who need a lightweight but powerful programming model — not a library of predefined prompts and integrations.

It’s a very different experience from the hand-holding of LangChain, but it packs reusable magic in generic constructs like annotate, compile, etc that work with arbitrary programs.

https://github.com/stanfordnlp/dsp/

I cannot praise Deepset Haystack enough for how simple they make things compared to LangChain, between the Preprocessor, the Reader/Retriever, and the PromptNode - the APIs, docs, and tutorials are quite easy to modify to your use-case.

Not affiliated, just a happy defector from LangChain.

I also was underwhelmed by langchain, and started implementing my own "AIPL" (Array-Inspired Pipeline Language) which turns these "chains" into straightforward, linear scripts. It's very early days but already it feels like the right direction for experimenting with this stuff. (I'm looking for collaborators if anyone is interested!)

https://github.com/saulpw/aipl

As someone who has created several LLM-based applications running in production, my personal experience with langchain has been that it is too high of an abstraction for steps that in the end are actually fairly simple.

And as soon as you want to slightly modify something to better accomodate your use-case, you are trapped in layers & layers of Python boiler plate code and unnecessary abstractions.

Maybe our llm applications haven’t been complex enough to warrent the use of langchain, but if that’s the case, then I wonder how many of such complex applications actually exist today.

-> Anyways, I came away feeling quite let down by the hype.

For my own personal workflow, a more “hackable” architecture would be much more valuable. Totally fine if that means it’s less “general”. As a comparison, I remember the early days of HugginfaceTransformers where they did not try to create a 100% high-level general abstraction on top of every conceivable Neural Network architecture. Instead, each model architecture was somewhat separate from one another, making it much easier to “hack” it.

I'm glad I wasn't the only one that felt Langchain had a ton of redundant abstractions engineered to gain clout for vc money. Here is an example:

AnalyzeDocumentChain[1] just wraps RecursiveCharacterTextSplitter[2]. It serves no real purpose except padding the api doc.

[1] https://js.langchain.com/docs/modules/chains/other_chains/an... [2] https://js.langchain.com/docs/modules/chains/other_chains/su...

If you know little about prompt engineering and want to throw together a demo of something that kind of works extremely quickly, or experiment with an LLM agent exactly as it's defined in some paper, LangChain is pretty useful.

If you want to develop a real LLM application, you're probably better off skipping the library completely, or at least fully understand each abstraction to make sure it does everything you want before you decide you want to incorporate it.

I’ll repeat what I’ve said in another thread the other day —

To put together a basic question/answer demo that didn't quite fit the LangChain templates, I had to hunt a bunch of doc pages and and cobble together snippets from multiple notebooks. Sure, the final result was under 30 lines of code, BUT: It uses fns/classes like `load_qa_with_sources_chain` and `ConversationalRetrievalChain`, and to know what these do under the hood, I tried stepping into the debugger, and it was a nightmare of call after call up and down the object hierarchy. They have verbose mode so you can see what prompts are being generated, but there is more to it than just the prompts. I had to spend several hours piecing together a simple flat recipe based on this object hierarchy hunting.

It very much feels like what happened with PyTorch Lightning -- sure, you can accomplish things with "just a few lines of code", but now everything is in one giant function, and you have to understand all the settings. If you ever want to do something different, good luck digging into their code -- I've been there, for example trying to implement a version of k-fold cross-validation: again, an object-hierarchy mess.

Given that the company has $200 million valuation, that is $2 million per line of code! just kidding.

Still, I would like to understand $200 million valuation of langchain.ai.

For me Langchain is glue code between a lot of commonly used LLM building blocks and prompts.

It is great to get a prototype 80% of the way there fast in order to validate an idea or run something short lived.

I suspect that, if you want to go further (simpler code, better control message length, reliability, etc), you will be better served by implementing the functionality you need yourself.

For the calculator tool I suggest instead to just generate Javascript as an output with temperature set to 0 (system prompt set to something along the lines of: "Generate native Javascript code only. Don't provide any explanations. Don't import any extraneous libraries") and then eval that Javascript code in a VM. Deno is a good candidate for this as it has good security settings with access to filesystem and network turned off by default. You can use something like deno-vm [1] to execute it separate from your running process too. Setting GPT-4 as model works even better. I have seen it perform better than Wolfram Alpha in many cases so I am wondering why OpenAI chose to integrate with Wolfram Alpha for this. GPT-4 was able to solve some really complex math problems I threw at it.

[1]: https://www.npmjs.com/package/deno-vm

Personal experience: was using LangChain and its output parsers for getting structured data. It was having a very high error rates (probably prompt was becoming too long and confusing). But it is a just prompt + some parsing logic. Replaced it with straight asking openAI GPT for json that matches some Rust struct / Python data-class. The errors went down and got one extra dependency out from the project. Tried to use its self hosted embeddings but the implementation (strangely) seemed tied to something called Run-house.

Not to belittle the library, but most of it is a very thin wrapper classes that reek of premature abstraction, couple with hit-n-miss docs. At this point, given the hype, it is primarily optimized for cooking up demoes quickly. But not sure if the valuations or production use is justified.

There are a few ways to use Langchain. Firstly, the docs are a mess. What I personally did, I followed the notebook from OpenAI cookbook on embedding a code base, and one on embedding the docs, and querying over that with GPT-4.

After a while of doing that, I realised like many others that it's too high of an abstraction. In the end I think you're better off just looking at their source code, and just looking at how they've implemented the stuff in normal python and then adapting it for your own needs.

Hmm, this is great food for thought! I am working on a Haskell based REPL for GPT, called GPTi[1] which might benefit from this approach.

[1] https://github.com/mlang/gpti

Related comments about ReAct: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

We ported the core of LangChain to Ruby, and while it is way more that 100 lines, I would give similar feedback as the author. Here is the repo if anyone is interested https://github.com/BoxcarsAI/boxcars

The problem with all these new fields is that the first code that gets popular is from people who are good at marketing not those who are good at programming.

we're still in the stage of LLM adoption when we can have "eye-opening" simple discoveries weekly. Langchain has momentum because of this, as a library of simple ideas. this period will end, and if they don't figure out the next step they're gone

The same goes for gpt-index.