Show HN: GPTCache – Redis for LLMs | Hacker News Reader

Show HN: GPTCache – Redis for LLMs

github.com

7 pointsfzliu5 comments

Hey folks,

As much as we love GPT-4, it's expensive and can be slow at times. That's why we built GPTCache - a semantic cache for autoregressive LMs - atop the vector database Milvus and SQLite.

GPTCache provides several benefits: 1) reduced expenses due to minimizing the number of requests and tokens sent to the LLM service 2) enhanced performance by fetching cached query results directly 3) improved scalability and availability by avoiding rate limits, and 4) a flexible development environment that allows developers to verify their application's features without connecting to LLM APIs or network.

Come check it out! https://github.com/zilliztech/gptcache

Open articleView on HN

Smart:

> However, using an exact match approach for LLM caches is less effective due to the complexity and variability of LLM queries, resulting in a low cache hit rate. To address this issue, GPTCache adopt alternative strategies like semantic caching. Semantic caching identifies and stores similar or related queries, thereby increasing cache hit probability and enhancing overall caching efficiency.

> GPTCache employs embedding algorithms to convert queries into embeddings and uses a vector store for similarity search on these embeddings. This process allows GPTCache to identify and retrieve similar or related queries from the cache storage, as illustrated in the Modules section.

While I wont pretend to fully understand this I assume it means more cache hits for similar but not exactly the same queries?

You are right, most LLM queries won't be exact match, so you will need semantic search for hit those similar but not same questions

Nice idea, thanks for publishing this, wondered for a long time why this wasn't yet in something like langchain (put it down to the storage aspect, despite query/query_with_sources being released recently). Having a cache system like this for the last 2 years we ran GPT-3 before the hype came has prevented us switching to langchain.

You use a vector database for similarity search right, which vector DB do you use, do you your product ziliz?

The current implementations use Milvus and FAISS, but we have opens for other open-source vector databases as well.

Reply on news.ycombinator.com