Hacker News Reader

Congrats on the traffic, I hope you get a high conversion rate :)

I assume you do more than one GPT call per search, e.g. rewriting user's query into multiple search queries, summarization of the results, etc.?

It's complex. We make a lot of GPT calls. Lots of them. And, for lots of different purposes.

How are you handling the unit costs, rate limits, and model risk?

That’s a deep question. Under the hood we use some LLM gateways like Helicone and https://usevelvet.com to track cost per request and optimize it.

Update: >5k requests per minute to OpenAI right now

Don't forget to rate limit every end point.

Yup, we have some decent safeguards in place. It’s technically difficult to allow logged out users to try the product, but I think people appreciate it.