Retrieval in LangChain
blog.langchain.dev
blog.langchain.dev
The link below is the transcript of a session in which I asked the agent to create a hello world script and executes it. The only input I provide is on line 17. Everything else is the langchain agent iteratively taking an action, observing the results and deciding the next action to take.
This is just scratching the surface. I've seen it do some crazy stuff with the AWS CLI. And this is just with GPT-3.5, I don't have access to GPT-4 yet and it clearly has better capabilities.
I just started tinkering with terraform, which it seems to understand fairly well.
I'd say that a person using Linux for the first time might do what is happening in that demo.
Anyway, not to diminish the bot or LLMs or whatever, but it's definitely something that could be done using the Stack Overflow Search API or similar. I'd say it's actually a very primitive / boring use of an LLM.
#!/home/ubuntu/venv/bin/python3.10
from langchain.agents import load_tools
from langchain.agents import initialize_agent
from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(model='gpt-3.5-turbo',temperature=0)
tools = load_tools(['python_repl', 'requests', 'terminal', 'wolfram-alpha', 'serpapi', 'wikipedia', 'human', 'pal-math', 'pal-colored-objects'], llm=llm)
agent = initialize_agent(tools, llm, agent="chat-zero-shot-react-description", verbose=True)
agent.run("Ask the human what they want to do")
For example, I asked it to write conway's game of life, and it took about 4-5 attempts but it wrote fully functional code that popped up a matplotlib window with a fully functional simulation. This would've taken me a day at least.
I asked it to write a FastAPI backend that uses SQLite for storing blog Posts, and it struggled with that one a lot and couldn't quite get it right, although I think that's largely a limitation of the python REPL from langchain as opposed to GPT.
On the one hand I'm excited to build all sorts of new things and projects with this, but on the other hand I'm worried my standard of living will decline because my skills will become super commodified :/
It would be great if it also summarized what the error was, what was the fix, and how to run the code that it created. That’s all in the output but could be pulled out at the end.
You can ask it to summarize things if you like. It sometimes forgets to do so, however.
https://raw.githubusercontent.com/jla/gpt-shell/assets/examp...
That said, even gpt-3.5 will try multiple routes to get to the same endpoint. It seems to get distracted pretty easily though.
That’s true, gpt-4 is way more easy to guide with the system messages and it doesn’t forget the instructions as the conversation goes on.
I couldn’t figure out a gpt-3 prompt that could handle “This text is written in French” correctly (it thinks it’s written in French), but with gpt-4 you can include in the prompt to disregard what the text says and focus on the words and grammar that it uses.
As an example, I've been trying to use it to learn Zig since the official docs are ... spartan. And I've said, "here's my code, here's the error, what's wrong with it?" and it will go completely off the rails suggesting fixes that don't do anything (or are themselves wrong).
In my case, understanding/fixing the code would have required GPT-4 to know the difference between allocating on the stack/heap and the lifetimes of pointers. It never even approached the right solution.
I haven't yet gotten it to help me in even a single instance. Every suggestion is wrong or won't compile, and it can't reason through the errors iteratively to find a fix. I'm sure this has to do with a small sample of Zig code in its training set, but I reckon an expert C coder could have spotted the bug instantly.
It’ll be much better on subjects where there is too much information on the public internet for a person to efficiently manage and sift through.
But it most certainly did not.
I used GPT 3.0 to maintain a code library in 4 languages, I'd write Dart (basically JS, so GPT knows it well), then give it a C++ equivalent of a function I had previously translated, and it could do any C++ from there.
2. GPT4's training data likely doesn't include significant Zig use, since large parts of its training data cut off a few years ago. I use Rust and it doesn't know about any recently added Rust features, either.
This has interesting implications because it means people will gravitate towards languages/frameworks/libraries that GPT knows well, which means even less training data will be generated for the new stuff. This is a form of value lock-in.
That's the kind of problem that most people are just failing to see. The usage of this models might not in itself be problematic, but the changes that it bring are often unexpected and too deep for us to see clearly now. And yet, people are rushing towards them at full speed.
That said its really good at regurgitating stuff from StackOverflow. But once you step beyond anything that someone has previously done and posted to the Internet, it quickly gets out of its depth.
Some "agents" in their vernacular that I've built.
* A reminder system that can take a completely free-form English description and turn it into a precise date and time and schedule it with an external scheduler.
* A tool that can take math either in English or ascii like y = 2x^2 + ln(x) and turn it into rendered LaTeX.
* A line editor that let's you ingest Word documents and suggests edits for specific sections.
* A chess engine.
Like it's crazy at just how trivial all this stuff is to build.
I agree that some things really are trivial to implement and I think this opens the door to non-programmers who know a little Python or JavaScipt to scratch their own itches and build highly personalized systems.
Thanks for writing this.
Are any of those open source / are they on Github? Could you link it?
- We integrate with vector db's + ChatGPT Retrieval Plugin
- Submitted a Retrieval PR to langchain here: https://github.com/hwchase17/langchain/pull/2014
- would love to explore further integrations as a plugin in any outer agent system
My plan was to use https://github.com/cocktailpeanut/dalai with the alpaca model then somehow use llamaindex to input my dataset - a slack export. But it's not too clear how to train the alpaca model.
Another idea I've had is to "overfit" a generative model like GPT on a dataset but pay more attention to how url and the like are tokenised
Here you go https://twitter.com/theseamouse/status/1614453236349693953
That's what hypothetical embeddings solve: https://summarity.com/hyde
There are also encoding schemes for question-answer retrieval (e.g. ColBERT)
Also would you just return a list of likely candidates and loop over the result set to see if any info is relevant to the question and then have the the final pass try to answer the question.
If the embeddings are worth their salt, then they should not be influenced by paraphrasing with different words. Try the OpenAI embeddings or sbert.net embedding models.
https://huggingface.co/sentence-transformers/multi-qa-MiniLM...
You have late-interaction models, which replace the dot product with a few transformer layers and are able to learn complex semantics.
Of course this would adversely affect latency and embedding size, so you might want to compress and cache the answers, hence (shameless plug):
[0] https://python.langchain.com/en/latest/modules/chains/index_...
Haystack seems more polished for NLP tasks, but LangChain looks more extensible long term?
Thanks!
- LangChain's Retriever is analogous to ChatGPT Retrieval Plugin. - In general, LangChain has tools for what ChatGPT calls Plugins. - ChatGPT uses OpenAI's GPT-4 LLM. LangChain uses ... any LLM (i.e. configurable).