Hacker News Reader

Chain of thought is underutilized. It almost never makes sense to show the user the "bare" response of the LLM. It's so easy to have LLMs self-critique, think through user intent, etc. to drastically improve the final output

View on HN

Improve probably, but speed also matter, I worry a lot about chaining lots of LLM calls together, each of them takes a bunch of seconds, and then the experience becomes just really slow.

You don't make multiple calls, you ask for a structured output and use some of the keys as the "chain links" in the chain of thought

Is there an example on how to do this ?

Contrived example, but say I'm summarizing an arbitrary document, I might ask for JSON with the following keys and instructions:

- purpose: one sentence on why would someone want to summarize this document

- effect: one sentence on how this affects the strategy we should use to summarize it vs a naive approach

- summary: one paragraph with the summary of the document based on purpose and effect

The LLM will change its summary based on the keys that come before summary in a meaningful way

Yeah you better be 99.9% perfect if you are going to take 60 seconds of plain spinner time to come back to me.

Which model are you using in production that's so slow you'd be waiting for 10 seconds, let alone 60?

Reply on news.ycombinator.com