Hacker News Reader

Full thread

LoganDark

How is it possible for the LLM completion to be so fast? I can't even run 169M RNN models that quickly.

View on HN

Oh, it just uses OpenAI.

https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fa...

...and has a 50-request limit per day.

https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fa...

...and trims the content to, um. 500 characters?!

https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fa...

So past a certain point you'll get the same completion repeatedly / a completion of the start of your document, cut off.

I think the nature/feel of the quick response is likely due to the fact the completion call is streaming "stream: true" [1]. Then, they are parsing it back to text as it streams the response back.

[1] https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fa...

I'd have been more surprised if there wasn't streaming. All the LLMs I have ever used, stream tokens in real-time. I'm talking about the time spent per token.

Oh yeah good catch – just bumped up the character limit to 5,000 (forgot I just upgraded to gpt-3.5-turbo-16k today) :)

Reply on news.ycombinator.com