Cool stuff! I suspect we will now see a bunch of startups coming up focused on helping companies reduce their AI usage, similar to how we have companies focused on optimizing cloud costs.
26 karma · joined October 17, 2018
So over the past few years, I have seen how contexts have been steadily growing in AI apps. And while the context lengths of LLMs have also been increasing, they are still effectively about 200k tokens. The performance drops off a cliff after that (you might have noticed it as well with long AI chats).
It is a simple API that prunes away irrelevant parts of a context for a given prompt, a.k.a. context-aware pruning. Integration is super simple: just an extra API call before the final LLM API call. You can get an API from the website.
I would love to chat if this is something that is relevant to you and if you have any feedback on what we are building!
I realised that I'm a horrible prompt writer and so are many other people. So i created this extension to help me write better prompts, using templates, save prompts, and optimize it for better performance. No login, not paid, just a useful little extension. Check it out!
All resources are free or very cheap.
Help support your fellow laid-off workers by joining as a mentor!