It seems to me that this can be easily added ad-hoc by programmers in their projects. But the biggest reason I see may be standardization of some kind? For interoperable building blocks maybe
Langchain is a swiss army knife of tools. And while that can be a good thing, it's mostly a bad thing when you're trying to build something for production. It's also not very flexible... when you're trying to do anything beyond the pre-built stuff, you have to edit the package itself... which means maintaining your own forked version or submitting a PR with your changes. For example, I improved upon the main text splitter Langchain recommends (https://github.com/ShelbyJenkins/langchain/blob/ca14d3028a57...), but I just haven't gotten around to doing everything required for a PR to add the feature (tests, docs, notebooks). So now I have two repos to maintain...
But I also really don't want to take the time to set up a web scraper for a dozen different document sources when Langchain has tools that will work with a minimal amount of tweaking. So, I'm happy to use it for things like that, and happy to recommend it for it's community built pocket-knife tools!
It’s simple: a bit of cargo culting and the real advantage that LangChain provides in its interoperability with various other useful systems.
In addition, when it comes to prototyping for a specific use case, we found it is often more than just calling the model but also the orchestration process matters, for example, when should LLM agent stop answering questions, fix input argument, ask a custom clarifying questions and more.
Hope AutoChain makes your exploration easier and more robust!
So maybe a simpler library like Microsoft's Guidance (https://github.com/microsoft/guidance)? It does this really well.
Also LangChain has a lot of integrations, that just pop up as soon as new API for anything LLM pops up, so that helps with new user onboarding as well.
Kind of a chicken-and-egg problem (to be widely used, it helps to be simple, but not all simple-to-use products make it to being widely used). TensorFlow vs PyTorch anyone?
Now, which will be more likely to be widely adopted? No one knows, just keep using both, learning both linguistics and hopefully one of them gets enough traction and all your effort put into building something with these two doesn't go to waste, fingers crossed.