kreidema

Do I understand correctly that your service works as a registered device into my whatsapp account with full "admin" like access? This seems a bit harsh for the sensitive data that peoples wahatsapp contain? Or am I missing a usecase here where that is unproblematic? Business accounts maybe?

I completely forgot that AI can very much also attack networks/devices in the wild. Interesting project.

I actually like this one. To me this feels like whatever AgentCore runtime runs on. Seems very usable.

But I guess for Agents I would continue using AgentCore.

Maybe if I wanted to build my own AgentCore though.

Fair question! The main thing to think about here: We are not caching responses, we are caching intermediate calculation results (see first graphic in the post). And those are most of the time just the result of a part of the full prompt. So maybe just the system prompt, but then not what comes after, etc.

This way we can save Anthropic and co. for example the effort of recalculating all the linear algebra needed for the prefill of the system prompt, for which they reward us with reduced input token cost. The result is the same, if cached or not cached, it's just less computation.

For the same prompt you would still get the same answer (assuming temperature = 0).

The big savings you will get in an agentic/conversational context: Every new turn always puts the full message array back into the request. If we don't cache the calculation result at every step, the provider has to recalculate the early turns potentially hundreds of times (see second/third graphic).

This is annoying because tools like conductor use the SDK. So this will either be the end of conductor for me or I switch to codex. Interesting dilemma.

Been using BetterAuth for half a year/one year now. It's exactly what I always wanted when using Clerk (and its free as a side effect). I was never a fan of the "we'll manage your user Table", always set up the Web hooks to sync everything to my own user table, but still loved the development experience clerk provided (most of it).

I will never go back. Lets see if in 3 years I have a different opinion, but I can't imagine so.