Are there any beginner-friendly Python packages that you would recommend to facilitate fast experimentation with such ideas?
For your example; Write a python script with requests that hits the OpenAI API. You can even hardcode the API key because its just a script on your computer! Now you have the GPT-4proLight-mini-deluxe response in JSON. You can pipe that into a bazzillion and one different places including another API request to Anthropic. Once that returns, you can now have TWO llm responses to analyze.
I tried haystack, langchain, txtai, langroid, CrewAI, Autogen, and more that I am forgetting. One day while I was reading r/Localllama someone wrote; "All these packages are TRASH, just write python!"... Lightbulb moment for me. Duh! Now I don't need to learn a massive framework to only use 1/363802983th of it while cursing that I can't figure out how to make it do what I want it to do.
Just write python. I tell you that has been massive for my usage of these LLM's outside of the chat interfaces like LibreChat and OpenWebUI. You can even have claude or deepseek write the script for you. That often gets me within striking distance of what I really want to achieve at that moment.
I've settled on the Mirascope library (https://mirascope.com/), which suits my use cases and lets me implement structured inputs/outputs via pydantic models, which is nice. I really like using it, and the team behind it is really responsive and helpful.
That being said, Pydantic just released an AI library of their own (https://ai.pydantic.dev/) that I haven't checked out, but I'd love to hear from someone who has! Given their track record, it's certainly worth keeping an eye on.
In langroid you set up a ChatAgent class which encapsulates an LLM-interface plus any state you'd like. There's a Task class that wraps an Agent and allows inter-agent communication and tool-handling. We have devs who've found our framework easy to understand and extend for their purposes, and some companies are using it in production (some have endorsed us publicly). A quick tour gives a flavor of Langroid: https://langroid.github.io/langroid/tutorials/langroid-tour/
Feel free to drop into our discord for help.
I'm working on a naive approach to identify errors in LLM responses which I talk about at https://news.ycombinator.com/item?id=42313401#42313990, which can be used to scrutinize responses. It's written in Javascript though, but you will be able to create a new chat by calling a http endpoint.
I'm hoping to have the system in place in a couple of weeks.
https://github.com/jackmpcollins/magentic
It's based on pydantic and aims to make writing LLM queries as easy/compact as possible by using type annotations, including for structured outputs and streaming. If you use it please reach out!
I tried many python frameworks but the lack of customizations and observability limited the utility. Now, I only use Instructor (Jason Liu's library). That, and concurrent futures for parallel processing.