Zep: Fast, scalable building blocks for production LLM apps | Hacker News Reader

Zep: Fast, scalable building blocks for production LLM apps

github.com

104 pointsroseway413 comments

Interesting project, thanks for submitting it!

Obviously not commenting on this project as I’ve yet to read up on it, but what are your proven to be useful building blocks for LLM apps? This space is super new and everyone is experimenting, promoting their solutions and trying to build communities/stargazers/customers around them. It is getting hard to distinguish what’s just fluff and what is an actual improvement over raw OpenAI APIs.

Even the more popular projects such as LangChain and TypeChat are a bit hard to rate regarding whether they’re actually quality higher level libraries worth adopting or if I should just read their source as inspiration and build my API calls myself.

I would really appreciate suggestions!

Great question. IMHO it's better to write something yourself. These things are not complicated and you can have better control and understanding by writing it yourself. Once the industry settles on a set of tools, you can start thinking of migrating if needed.

Zep has Python and TypeScript packages for teams building without LangChain: https://docs.getzep.com/sdk/

Zep author here. LangChain is a great framework and has a very broad and active ecosystem. Many of LangChain's core components, such as chat history "memory," history summarization, entity extraction, vector search, and more, don't scale well in production. They operate in memory and often synchronously within the chat loop, resulting in poor user experiences and limiting deployment options.

You can use third-party ecosystem integrations with external SQL databases, vector databases, etc to fix some of these issues. This requires some know-how, infrastructure, and time. Zep's building blocks are turn-key solutions to these challenges and offered in a single service.

The Zep GitHub project, website, and demo video provide a good overview of the project's functionality and how the service solves these issues.

This blog post on the LangChain website offers some benchmark data using Zep vs LangChain's core memory components: https://blog.langchain.dev/zep-x-langchain-slow-chatbots/

Since you are targeting developers I would highlight the differentiating features: "database-backed", and "asynchronous", or whatever they are. This is more informative than "production-grade", "fast", or "scalable", which anyone can claim. In fact, the competitors have a better claim to being fast -- they're in memory! -- so I would drop them unless I had benchmarks. Something I would also encourage you to work on.

Finally, call out your competitors, and explain your value add for each one. Maybe in a table. Why not just use a vector database and hand roll the rest, for example?

Good luck!

Thanks for the advice! Some benchmarks here: https://blog.langchain.dev/zep-x-langchain-slow-chatbots/

That's not really benchmarks. To me, it just shows why putting things like langchain in the mix makes your life harder in the long run. Having to add more tools is not an ideal solution. What people want are actual "building blocks" that don't require adopting a framework

Is this building blocks or a platform?

When someone tells me "building block" I would expect to be able to use just one of the pieces in my application without a lot of extra fuss.

This looks way more like a framework and platform, with multiple deployments needed to just get started, and a cloud offering in the works.

If this is a platform, what makes it different from all the other platforms? (besides prefixing the library functions with Zep, like ZepReader and ZepStorage)

Convince me to not just use Postgres for everything!

this

You could certainly use Postgres for everything. Zep does. But then you'd also have to build the stuff that Zep does, too.

The video in the github repo is very helpful. I think it needs an example of what can be built with Zep though, not just the features.

Thanks. I definitely want to make a longer video demonstrating Zep end-to-end.

Reply on news.ycombinator.com