714 karma · joined May 5, 2016
hn@zak.knill.dev
Analogies about 'students and exams' that don't quite fit. The sentences ending in artificially punchy short phrases. The 'correction' and dash-as-reveal in "they can't answer it - no one can".
It's not good, or a compelling read, and I think it's lazy not to have swept through and fixed these things.
This is classic Claude, and makes me think the author didn't read, edit, or invest in their own words.
When you make an LLM call, if you use the streaming responses (typical of token streaming into applications) then the response from the LLM provider comes in a specific format; mostly made up of individual json events, one for each text-delta or token generated, and some framing around start/stop. That API format differs across different LLM providers, and also isn't that convenient to process for what you actually want to show in the client/browser.
The backend portion of these sdks (AI-SDK) are an abstraction over the LLM provider, and re-write the 'events' that you get back from the LLM provider into an abstracted json format that's a little bit easier to work with. And not coupled to a specific provider.
The frontend portion of these SDKs (AI UI SDK) helps you to render those message into a frontend interface, and uses the same abstracted format that the backend uses.
There's one big missing piece though, and that's the fact that the transport between frontend and backend is over a single HTTP+SSE stream. If that connection fails, all the work that the backend is doing cannot be transported to the frontend anymore. The HTTP connection also doesn't allow for useful features like cancellation, steering, interruption, and multiple devices.
If you want those, checkout Ably AI Transport, which targeting the _transport_ section between backend and frontend, to make sure you get reliable experiences in the UI regardless of which AI provider or SDKs you use.
Disclaimer, I work for ably on this product:
I think they really missed the opportunity to add in resumable streaming after a dropped SSE connection, or multi-device support. Because these SDKs are easy to drop-in to your applications, but hard once you actually try and build resilient systems on top of them. Systems that can heal from dropped connections, or support multiple users or multiple devices.
There's a lazy habit from some folks to say something they either disagree with or don't understand was "written by AI" without backing up that statement.
So you can be notified by the database, but you can't (with the stateless HTTP + loadbalancer design explained in the article) get that notification back to the client. Because the client isn't connected anymore; so how does the client know that there's new information?
Please, please, please don't say "polling". Because you've clearly missed the entire argument of the article if you say polling.
It's a bit harder to do agent presence ('is the agent still there') with this model without heartbeats, but possible.
It's good to see the industry starting to address the "durable sessions" problem, because it sucks.
100% - the argument of the article is that building any feature beyond chat-based-demos on HTTP SSE streaming is super complex. But a lot of folks still want to do it, because that's what their tech stack is. I think it's still a valuable thing to be talking about how you might do that.
This post says little about that, and suggests some improvements the _reviewer_ can make.
I think that's completely the wrong end of the stick to be tackling. As the burden is still on the reviewer and not the author.
> The OAuth consent screen will give you a choice of three levels of access: read-only (see emails, contacts, calendars), write (update emails, save drafts, edit contacts and events), and send (send emails).
If you look at the gifs of the Claude UI in this post[1], you can see how the HTTP response is broken on page refresh, but some time later the full response is available again because it's now being served 'in full' from the database.
[1]: https://zknill.io/posts/chatbots-worst-enemy-is-page-refresh...
This is absolutely the hardest bit.
I guess the short-cut is to include all the chat conversation history, and then if the history contains "do X" followed by "no actually do Y instead", then the LLM can figure that out. But isn't it fairly tricky for the agent harness to figure that out, to work out relevancy, and to work out what context to keep? Perhaps this is why the industry defaults to concatenating messages into a conversation stream?
Obviously polling works, it's used in lots of systems. But I guess I am arguing that we can do better than polling, both in terms of user experience, and the complexity of what you have to build to make it work.
If your long running operations just have a single simple output, then polling for them might be a great solution. But streaming LLM responses (by nature of being made up of lots of individual tokens) makes the polling design a bit more gross than it really needs to be. Which is where the idea of 'sessions' comes in.
The pattern I describe in the article of 'channels' works really well for one of the hardest bits of using a durable execution tool like Temporal. If your workflow step is long running, or async, it's often hard to 'signal' the result of the step out to some frontend client. But using channels or sessions like in the article it becomes super easy because you can write the result to the channel and it's sent in realtime to the subscribed client. No HTTP polling for results, or anything like that.
Of course the hard bit then is; how does the client know there's new information from the agent, or a new session?
Generally we'd recommend having a separate kind of 'notification' or 'control' pub/sub channel that clients always subscribe to to be notified of new 'sessions'. Then they can subscribe to the new session based purely on knowing the session name.
The article is about how agents are getting more and more async features, because that's what makes them useful and interesting. And how the standard HTTP based SSE streaming of response tokens is hard to make work when agents are async.
With this change, it looks like an officially sanctioned version of *claws. Connecting to whatever "channels" you want via MCP.
Architecturally it's a little different, most *claws would call the Agent SDK from some orchestrator, but with claude channels the claude code binary starts the MCP server used to communicate with the channel. So it's a full inversion of control where Claude code is the driver, instead of your orchestrator code.
I updated my nanoclaw fork to start the claude code binary in a docker container on PID 1, and you can read the docker logs straight from claude code stdout, but with comms directly to/from your channel of choice. It's pretty neat.
Querying for something like "snowflake json from variant?" in both engines and in google I get a sort-of-right-but-not-really-that-helpful ai summary about "parse_json" function. In Kagi I get an actually useful summary with code examples of parse_json, but also the colon-based syntax for accessing values inside nested objects without needing to parse anything.
I very rarely need to go into a page, I use Kagi quick search summary with the "?" suffix and it almost always gives me a useful answer in one-shot.
What you should do, is use the context window that you've got from writing the code and refine that into a commit message using a skill.
Without the skill, the commit messages are often a list of files changed with a list of bullet points at the end saying:
- Added 8 tests to file.xyz
Pretty useless for a reviewer, they can see that from the diff.
Or is it that all your tokens go through a DB anyway?
It's fairly easy to keep an agent alive when a client goes away. It's a lot harder to attach the client back to that agents output when the client returns, without stuffing every token though the database.
The article touches very briefly on Phoenix LiveView and Websockets. I wrote about why chatbots hate page refresh[1], and it's not solved by just swapping to Websockets. By far the best mechanism is pub/sub, especially when you can get multi-user/multi-device, conversation hand-off, re-connection, history resumes, and token compaction basically for free from the transport.
1: https://zknill.io/posts/chatbots-worst-enemy-is-page-refresh...