Hacker News Reader

> although time to first token is... not great

The token cache works on CUDA too. So yeah, the initial loadout sucks, but almost everything from then on is solid.

This works for scenarios like a simple chat. But the moment you start throwing large documents on it, or lots of code, it's back to waiting.