Langchain Is Pointless

The reason why Langchain is pointless is that it's trying to solve problems on top of technical foundations that just cannot support it.

The #1 learning is that there is no reusability with the current generation of LLMs. We're using GPT-4 and 3.5T exclusively.

Over the last several months, my team has been building several features using highly sophisticated LLM chains that do all manner of reasoning. The ultimate outputs are very human-like to the point where there is some private excitement that we've built an AGI.

Each feature requires very custom handwritten prompts. Each step in the chain requires handwritten prompts. The input data has to be formatted a very specific way to generate good outputs for that feature/chain step. The part around setting up a DAG orchestration to run these chains is like 5% of the work. 95% is really just in the prompt tuning and data serialization formats.

None of this stuff is reusable. Langchain is attempting to set up abstractions to reuse everything. But what we end up with a mediocre DAG framework where all the instructions/data passing through is just garbage. The longer the chain, the more garbage you find at the output.

We briefly made our own internal Langchain. We tore it down now. Again not that our library or Langchain was bad engineering. It's just not feasible on top of the foundation models we have right now.

I have a full-on "The Problem With LangChain" blog post in the pipeline, and the reason I made a simple alternative (https://news.ycombinator.com/item?id=36393782) because I spent a month working with LangChain and coming to the conclusion that it's just easier to make my own Python package than it is to hack LangChain to fit my needs.

A few bullet points:

- LangChain encourages tool lock-in for little developer benefit, as noted in the OP. There is no inherent advantage into using them, and some have suboptimal implementations.

- The current implementations of the ReAct workflow and prompt engineering are based on InstructGPT (text-davinci-003), and are extremely out of date compared to what you can do with ChatGPT/GPT-4.

- Debugging a LangChain error is near impossible, even with verbose=True.

- If you need anything outside the workflows in the documentation, it's extremely difficult to hack, even with Custom Agents.

- The documentation is missing a lot of relevant detail (e.g. the difference between Agent types) that you have to go diving into the codebase for.

- The extreme popularity of LangChain is warping the entire AI ecosystem around the workflows to the point of harming it. Recent releases by Hugging Face and OpenAI recontextualize themselves around LangChain's "it's just magic AI" to the point of hurting development and code clarity.

Part of the reason I'm hesitant to release said blog post is because I don't want to be that asshole who criticizes open source software that's operating in good faith.

I agree, I really don’t like LangChain abstractions, the chains they say are “composable” are not really, you spend more time trying to figure out langchain than actually building things with it, and it seems it’s not just me after talking to many people

Their code seems all rushed, and seems it worked out for initial popularity, but with their current abstractions I personally don’t think it’s a good long term framework to learn and adopt

That’s why I built my own alternative to it, I call it LiteChain, where the chains are actual composable monads, the code is async streamed by default not ducktaped, it’s very bare bones yet but I’m really putting effort on building a solid foundation first, and having a final simple abstractions for users that don’t get in the way, check it out:

https://github.com/rogeriochaves/litechain

It is pointless - LlamaIndex and LangChain are re-inventing ETL - why use them when you have robust technology already?

1. You ETL your documents into a vector database - you run this pipeline everyday to keep it up to date. You can run scalable, robust pipelines on Spark for this.

2. You have a streaming inference pipeline that has components that make API calls (agents) and between them transform data. This is Spark streaming.

Prophecy is working with large enterprises to implement generative AI use cases, but they don’t talk so much on HN.

Here’s our talk from Data+AI Summit: Build a Generative AI App on Enterprise Data in 13 Minutes https://www.youtube.com/watch?v=1exLfT-b-GM

Here’s a blog/demo https://www.prophecy.io/blog/prophecy-generative-ai-platform...

I have looked for the value and never really found it.

It seems to mostly be (bad) abstractions around things you could easily do without langchain.

Take one of the most important llm things: prompt templates. What does langchain add over a simple function and an f string? Maybe I'm missing the point, but I can't find anything.

Anyway, it seems people like it so who am I to judge, but I don't like making our codebase dependent on a huge new library with unnecessary abstractions and little or no value add.

Since last year, before I heard about langchain, I've been building my own stack of tooling for my own LLM projects that probably now covers about 10-20% of Langchain's functionality. I heard about Langchain earlier this year and groaned, thinking that I did a lot of work for nothing..

..Then I actually used langchain. I was shocked at how poorly performant the code is. Some operations took 10x longer than how I did it, and all the while producing worse results. As tempting as it is to just roll with langchain from day one, I'd highly advise against it. Think deeply about what you're actually trying to accomplish and instead of just injecting langchain in the middle of everything as this messy, amorphous glue code thing.

Remember, this is the project that raised ~$30m from Benchmark and Sequoia.

There was a controversial "quality doesn't matter for software products" post and discussion[0] here on HN a few days ago and this is a beautiful example.

Product may matter eventually, but you can sure surf the hype for a long time before the reckoning comes (and if you're lucky, you may even be able to get someone else to hold the bag then).

0: https://news.ycombinator.com/item?id=36615286

Using an LLM framework at this moment doesn’t make sense and can be damaging, in my humble opinion. Ways to extract value from LLMs are in early exploration stage. Look at research in prompting: chain of thought, react, reflection, tree of thoughts, zero vs few hot etc. Then completion vs conversational interfacing. Then memory management via vector databases and prompt expansion vs compression vs progressive summarization etc. All these are fairly recent developments. They are not abstractions worth cementing, this is search and creative phase. LLMs threw everything in the air, but the dust is far from settling. I think it’s important to recognize the phase we’re in and pick your weapon accordingly. You have to stay nimble and light, ready to experiment with a new idea that will come out next week. You should be hacking these things together by yourself. If you pick a framework at this stage know that the framework will have to pay the price of trying to cement things in the times of storm. And you’ll be a few steps behind. Of course this is my personal take.

Proper term is that it's a "false abstraction"...

It abstracts some work only to introduce introduce it's own API (which is ultimately more complicated, less documented, introduces limits and constraints, and comes with its own bugs and dev politics)...

But nobody asked me, keep using it

I have done a bit of research trying to figure out why anyone would use langchain. The main two reasons I’ve found are these:

1. Newbies that want to play with LLMs don’t know where to start or what the major building blocks even are. Despite the complaints here about documentation their getting started docs will walk you through the concepts. Going from total ignorance and confusion to now having a rough understanding of loading a prompt with chat history, using an embeddings database, calling a completions endpoint, etc. will make people feel accomplished. And then lang chain has earned some loyalty just because they were there for you first.

2. In the case that you don’t know which embedding db, AI host, or model you want to use you can quickly swap those in and out and measure the results. That means there’s little reason to complicate your back end code with lang chain (I’ve always just written my own abstraction layer to make this possible with very few lines of code). But for a python notebook it can make sense.

Sad to hear the news, I have 7 years of langchain experience on my resume, now what am I going to do!?

I had the same thought about Langchain, and it's essentially my same criticism of most wrapper libraries and SDKs. What are they actually doing that can't be just as easily done with straight up string manipulation and HTTP requests? Usually very little. ORMs might be one of the few exceptions in some cases.

Even so, ORMs sell a similar false promise to Langchain, which is that you can "easily" swap out the underlying thing; a migration that is rare in practice and almost always not that simple.

You might not need Langchain.

I believe the abstractions in Langchain are inherently flawed. The core problem resides in the composability of chains. While it offers a handy way to create prototypes, it becomes restricting when you desire to modify a specific element within the chain. The hierarchical design of chains in Langchain conceals the component you wish to alter and obscures the parts developers might want to adjust, making the process of experimenting and refining the pipeline difficult.

The optimal abstraction for LLM apps, in my view, should resemble a DAG or a state machine. This alternative exposes the distinct stages in the pipeline rather than masking them in a hierarchy. Yes, adopting this new abstraction might lead to more code but it offers superior control. It's hardly surprising that many users start prototyping with Langchain, but then, when ready, they clone the prompts and construct their own systems.

Fixing this fundamental issue would be very difficult. It would necessitate reworking the library from the ground up.

I don't know enough to agree that it's pointless, but I'd agree that when I looked at it I saw a lot of abstraction of already simple stuff (like the examples the post gives) and decided that for what I was doing it would be faster and easier to understand to just write my own python script. Though I can picture for very inexperienced developers the abstractions may be helpful short term?

LangChain co-founder here. There's lots of good feedback here (that also resonates with previous feedback) that we're working hard to address. On some key points:

- We genuinely appreciate all the thoughtful criticism and feedback. Our goal is to make it as easy as possible to build LLM applications (both prototypes and production-ready applications), and if we're falling short in an area we'd much prefer to hear it rather than not. We don't have the bandwidth to respond to all feedback directly, but we do (1) appreciate it, and (2) try to address it as quickly as possible.

- Documentation: we've heard this for a while now, and have been working to improve it. In the past ~3 weeks we've revamped our doc structure, changed the reference guide style, and worked on improving docstrings to some our more popular chains. However, there is a still a lot of ground to cover, and we'll keep on pushing. Feedback on which specific chains/components need better documentation is particularly helpful

- Customizability: we need to make it easy to customize prompts, chains, and agents. We're thinking of changes to more easily enable this - better documentation, more modular components. We'll up the priority of this.

- Other tooling: there are general difficulties in building LLM applications that aren't strictly related to langchain, such as debugging and testing. We're working on building separate tooling to assist with this that we hope to launch soon.

I've definitely got heartache here, and they merit criticism, but it is real

We need a lot of pluggability to support diff vendor LLMs and BYO LLMs in Louie.ai, so having langchain has been nice for helping code to interfaces vs vendor lockin. It definitely has growing pains - ex: sync & multithreading is important for us so we are generally coding around langchain while that smooths out. Likewise, we ended up building much of our conversational and multitool capabilities as custom libraries vs using theirs for similar quality reasons. We can't use any of the codegen capabilities because they are massive security holes so doing our own work there too.

If anyone is into that kind of work (backend, AI & web infra, ...), definitely hiring for core platform & cool customer projects here: louie.ai / Graphistry.com/careers

I keep going back to LangChain thinking it just hasn't found its legs yet, but every time I do I retreat exasperated. I don't find their abstractions useful or intuitive, and their documentation is woefully scattered and incomplete. Things are moving so quickly with LLMs that theirs is no easy task, but so far they haven't really cracked the nut of making LLM app development easier.

Langchain was useful to me personally for two reasons: using their prompt templates as a starting point for my own, and seeing how their “tools” were built to learn about good Python libraries to build my own tools with.

Viewed through this lens, LangChain was more a “sample codebase” than a library for me, and it was reasonably good for that.

I've built a few LLM-based projects now and quickly discovered that Langchain was overkill (and not even very good overkill) for my use cases. I thought it was just me, glad to hear it's not.

Coming from a frontend background this reminds me a lot of the frontend situation some years ago when it was super common to npm install the stupidest pointless packages.

Of course some people still do, but hard lessons were learned and all experienced people I know are a lot more mindful and cautious what dependencies they add.

It seems to me that this space, and maybe data science more broadly, is currently in that situation. Maybe it's the lack of coding skills, maybe it's the transition from one-off research notebooks to production applications, or maybe it's just the norm to have big do it all libraries (like pandas, scikit etc), idk, but I expect the same lessons will be learned.

For Langchain I wonder how they will keep up with changes in all the things they have their abstractions on. I guess having a huge community helps, but that doesn't help you with compatibility. It would not surprise me if that will get really ugly.

Claude 2.0 summary of this:

Here are a few key points from the Hacker News discussion on Langchain:

- Many commenters feel Langchain introduces unnecessary abstraction and indirection, making simple LLM tasks more complex than just using Python and APIs directly. The abstractions don't seem to provide much real benefit.

- There are critiques about Langchain's poor documentation, lack of customizability, and difficulty debugging. The rapid pace of updates is also seen as problematic.

- However, some point out benefits like easy swapping of components (models, vector stores, etc) and potential future improvements. A few see value in it for learning or quick prototypes.

- Overall sentiment seems quite negative on the usefulness of Langchain for real production systems, with many sharing they tried it but went back to coding LLM workflows directly. Some think it's hyped and reflects poor technical choices.

- But a counterpoint is that its popularity makes it a safe choice to build on initially, despite architectural issues. The crowd may offset other downsides.

In summary, the overall vibe of the discussion appears quite critical of Langchain's technical merit and production readiness, though there is some defense of its prototyping and educational potential.

Wholeheartedly agree—it adds layers of bloat and abstraction between you and the actual ReAct pattern, which can be trivially implemented in maybe 50 sloc.

Langchain is as pointless as one of those kitchen tools that specifically cuts egg into slices, but you could have done that with a knife.

Does it introduce more lines of code? Yes

Does it introduce features that you don't need to implement yourself? Yes

Does it make it easy to drop OpenAI? Yes

What's the problem?

Thank god, there is someone else who feels the same way. There are tons of modules and none of them is production ready.

It feels good for hobby / college projects, but would not use it for production.

The pace at which they release features is also scary.

I had been hearing these pains from Langchain users for quite a while. Suffice to say I think:

1. too many layers of OO abstractions are a liability in production contexts. I'm biased, but a more functional approach is a better way to model what's going on. It's easier to test, wrap a function with concerns, and therefore reason about.

2. as fast as the field is moving, the layers of abstractions actually hurt your ability to customize without really diving into the details of the framework, or requiring you to step outside it -- in which case, why use it?

Otherwise I definitely love the small amount of code you need to write to get an LLM application up with Langchain. However you read code more often than you write it, in which case this brevity is a trade-off. Would you prefer to reduce your time debugging a production outage? or building the application? There's no right answer, other than "it depends".

To that end - we've come up with a post showing how one might use Hamilton (https://github.com/dagWorks-Inc/hamilton) to easily create a workflow to ingest data into a vector database that I think has a great production story. https://open.substack.com/pub/dagworks/p/building-a-maintain...

Note: Hamilton can cover your MLOps as well as LLMOps needs; you'll invariably be connecting LLM applications with traditional data/ML pipelines because LLMs don't solve everything -- but that's a post for another day.

Every system I’ve seen for managing this kind of system has flaws, including the ones that I have written.

For instance scikit-learn implements excellent algorithms for model selection that would apply, in principle, to a model based on huggingface transformers that might take 2 hours to train. skl is a fast machine if memory fits in RAM on a single computer, but it is not up to task for multiple computers or anything mortal to a single process such as the computer bring turned off.

HF has model selection algorithms too, but not as nice. They don’t take the same kind of datasets as all so it would be a hassle to import my ski models into HF.

I have to be able to compare models generated with any kind of tools so I think I will build a universal model selection framework (builds and test models) but then you run into the problems langchain did where there is a lot of structure imposed and all sorts of quirks and performance losses because of that structure.

For instance my current skl selector wastes a lot of resources computing stuff from scratch over and over again and if the code were properly organized it could get the job done 3 times faster but the same trick wouldn’t work for every other experiment I might want to do.

So we are all running into hurdles and finding ways to jump over them, making a lot of mistakes because we are in a rush and don’t know better yet.

I tried to improve the type coverage of the JS package and I came to a lot of the same conclusions. It feels like a lot of unnecessary and poorly conceived indirection and abstraction. There’s basically just a lot more to learn especially if you’re just getting started.

It’s maybe useful as a repository of prompting techniques, but I found myself constantly monitoring the actual prompt text being generated rather than reading the code that produces it…

My read on lot of justifiable criticism of langchain is, that it is rooted in trying not to lose out the lead it established since Nov 2022. Pace of updates in LLM space have been staggering.

Langchain hasn’t invested enough in quality of lib. A good lib needs good deliberation to build right abstractions that add value, then “get out of the way“.

I've had a similar experience with LangChain. Initially, I was really impressed with it while developing an MVP, but as soon as I needed to add complexity or specific features, things started to go downhill.

A significant chunk of my time was spent navigating through LangChain's codebase, trying to make sense of missing or outdated documentation. Plus, there's a significant disparity between the features of the JS/TS and Python versions. Identifying which version supports which features can be a real challenge.

I don't want to sound overly critical, as I'm currently using it in production. However, I had to modify it so extensively that it barely resembles the original project anymore. Looking back, I can't help but think that adopting this library might have been a misstep. Perhaps we should have taken the time to create our own.

I've been using the JS version of langchain for a few months now, and despite there being a lot of valid criticism, (especially around the abstractions it provides) I'm still glad to be using it.

We get the benefits of a well used library, which means making certain changes is easy. For example, swapping our vector database was a one line change, as was swapping our cache provider. When OpenAI released GPT-4, we were able to change one parameter, and everything still just worked.

Sure, it's moving fast, and could use a lot better documentation. At this point, I've probably read the entire source code several times over. But when we start testing performance of different models, or decide that we need to add persistent replayability to chains of LLM calls, it should be pretty easy. These things matter to production applications.

I think that Generative AI applications is a config management problem. Think Prompts X Chains X LLMs. Your prompts wont work across everything and everything will break on model change. Coding this into ur classes is what everyone does.

I think the better answer is to declaratively pull out the prompts X chains as jsonnet code. Call it trauma & learnings from the K8s/Borg world. We have formats that have evolved as a result of millions of lines of code wrangling clusters/terraform/etc - so we decided to build a SDK over it.

that is what we did here - https://github.com/arakoodev/EdgeChains/releases/tag/0.2.0

EdgeChains is basically Generative AI prompt engineering modeled as config management.

What business problems are you solving with Langchain and LLMs?

I tried using langchain.js after a bit of hype and overwhelming dev support and incremental releases, but it became added complexity for no reason for our case.

OpenAI's NPM is much easier to use. A lot of what langchain promised out of the box didn't work like caching. Anyways it's still beta i believe

This guy's analysis is terrible. I literally built a retrieval augmented generation with memory in langchain last night (comments claim you can't do this)

Sure, langchain has poor documentation and useless helper functions. That doesn't mean that it's pointless.

I found that you need to dig into the actual code and debug it while it is running to see what is going on to actually use the library (both with Langchain and LlamaIndex). That's unfortunate, but it does show a path to get you to where you want to go and is probably faster in the long run than writing your own code because you can swap components (e.g., which LLM or which vector DB) in and out.

Langchain is an obvious VC / investor hustle by a handful of smart developers who are betting on the low sophistication of entrants to ML and Data science.

It doesn’t do much but it wraps some obvious functionality with method names and paper thing abstractions that speaks the language of people who don’t know Python beyond Jupiter Notebooks.

All power to them, they saw a gap and pounced.

For me the biggest benefit is just following the project and seeing what the community is doing with llms (It’s also not bad for quick proofs of concept).

That said especially in python it’s not that hard to reimplement things yourself in a cleaner way. Output parsing for agents was nice but with the function update from OpenAI it’s not really necessary (if you’re just using their API)

Didn’t they raise at a $200 million valuation

I don't trust them, because they never talk about their "tradeoffs". Every software has its tradeoff.

Everyone was so hyped up by this useless thing that I thought I was going crazy. Literally everything is done faster and easier and more readable and more maintainable doing it with a lines of Python yourself. Lot of hype with AI frameworks, especially this one.

If you'd like another option built on a foundation of open LLMs, open models and open source, check out txtai (https://github.com/neuml/txtai).

Disclaimer: I am the author of txtai

const THE_LEFT_HANDED_IDEAL_FOR_THE_EVEN_NUMBERS_OF_THE_NATURAL_NUMBERS_UNDER_MULTIPLICATION = 2

IMO it's a great educational tool to understand how to build AI apps at least at a high level. It provides a good mental model. Can someone name another single resource that achieves the same?

I agree it's not so easy to work with Langchain.

That's wy I built the simple LLM agents repo which you can understand in 5 minutes:

https://github.com/mpaepper/llm_agents

If you’re using JS/TS and want to have a nicer API for LLM calls (and a bit more), check out https://github.com/lgrammel/ai-utils.js

Langchain at times remind me of a Java API to send HTML from server to agent.

HTMLElement root = new HTMLElement(); // serialize as text and send root to output stream

I was like why not send the string “<html>” to the output stream.

Stop doing prompt ops.

https://pbs.twimg.com/media/FzZxxmIacAEBPof?format=png&name=...

there is an alternative that is production-grade - deepset haystack https://haystack.deepset.ai/

p.s. i am contributor so there could be bias

LangChain is a perfect example of unnecessary abstraction. You could build a much simpler and more composable library for working with LLMs simply using functions and function composition. This isn't rocket science.

At one point I got so frustrated with langchain that I wrote my own stuff in go: https://github.com/ryszard/agency

Completely agree. I had this exact thought last week. It's been helpful to go through their repo to see how they've done certain things, but it's a textbook leaky abstraction.

It’s technical rent seeking and premature abstraction.

100% agree. It is just a tactical way to SLOW LLM startups down with debugging hell. Highly would not recommend.

I want an off the shelf “talk to PDFs” solution and was hopeful langchain was it.

I agree that LangChain is pointless for experienced ML developers building products. For the rest, I disagree as just getting to the point where same observation can be made is worthwhile.

What is langchain?

the time you spend to understand langchain, is much more than the time you make it yourself

LangChain is a "utils" library

Change my mind.

the consistency and conventionality of the methods in the langchain library are very sporadic, imo

shivashankar

It's easy to hate on langchain. Sure, the architecture isn't the best but it's where the crowd is so it's a safe bet