Hacker News Reader

Comment by Waterluvian | Hacker News Reader

On the topic of LLMs and developing more adaptive abilities, I was thinking:

LLM latency is a real problem for interactivity. Especially if you want to talk to it or have it talk to another AI (eg. A video game environment).

When I talk to someone, they don’t wait for my whole sentence to be spoken before parsing and processing and formulating a response. Sometimes they’ll even interrupt me. Can LLMs eventually do this? Can it begin to prepare a smaller collection of “thoughts” as it is fed a streaming input? Or is this at odds with what is actually happening vs. the human brain analogy?

View on HN

One could bolt such a system on top of an LLM.

An LLM is "just" a document completer. Given text, it predicts the following text.

So there is nothing stopping you from bolting on a system which works like:

- Given the current streaming input, continually update a pool of N likely continuations (predict what the user will say)

- For each of those N likely continuations, pre-generate responses

- If the actual user continuation matches one of the continuations, use your pre-generated response

This is trading off compute for latency, as you won't use at least N-1 of those pre-generated responses.

The loops is also quite large right now. Put a lot in, get a lot out, put a little in, get a lot out. I suppose there is some work from OpenAI to have a well formed response in a certain number of tokens or something, I am not sure how the "document size" part of their chat completion works.

That's exactly what chess programs do when they think on your time.

What sort of latency are people seeing from LLMs? I type faster than an LLM outputs tokens, so predicting my continuation would be slower than waiting for me to type it. Most people speak faster.

An LLM that knows when it should interrupt me, as the parent comment mentions, would be really cool but I don't think it has the ability to determine when an interruption would be helpful. I'd be annoyed if I was asking "what is one plus three plus five" but the LLM interrupts with "1+3=4", for example.

Like humans, context matters.

“This reminds me of… that movie where uhhh… the big guy goes to death row and he like… he can help people with his special powers, like that mouse he names or.. uhhh…”

You’d probably interrupt me and it would be appropriate and welcomed.

Sure, but would it interrupt with auto-complete nonsense too soon or would it know when an interruption would help?

> This reminds me of

>> A cool spring day?

> that movie where uhhh

>> Star Wars is a movie.

I'm human, so I can understand when I have enough info for a good guess (The Green Mile?) but that's a much different skill than what an LLM does, right?

In principle, understanding when it is appropriate to interrupt is the same sort of problem as understanding when it is appropriate to use a specific tool or API call. Both are deviations from natural language that the model has to decide to employ based on the context of the input, so yes it should be very possible for LLMs to do it. You're asking if they would do it well or if they would instead get it completely wrong and interrupt you all the time with irrelevant nonsense - I'm going to say that depends entirely on implementation.

The bigger issue, in my opinion, is that current models running on current hardware at reasonable costs simply aren't performant enough to do the faster-than-speech prediction that is required to execute the concept well.

I'm getting sub 2 second inference on V100 for Flan-UL2(20B)

Latency is mostly a matter of hardware, model size and small tweaks. LLaMa running locally is very fast. And current LLMs could definitely be finetuned to interrupt conversations like you mention.

This excites me. The biggest issue I have with assistants is the awkward cadence of any interaction with them. I can’t wait to see this actually existing in practice.

I think we'll get there. Not just from a pure performance perspective but also in terms of model architecture. Or maybe it's rather going to be an architecture of models? I mean, the human brain isn't a single(-threaded) entity, either. Lots of things are happening in parallel, and there are parts of your brain that do complete words & sentences and the like and then there are other mechanisms that evaluate those completions and decide whether or not to say them out loud etc.

It seems that this problem has been somewhat tackled already (Apologies for the sensationalized video title): https://youtu.be/SCfCuLy4RJA?t=298

Here’s a project attempting to do just this!

https://github.com/yacineMTB/talk

This is awesome and exciting. Thanks for sharing. The demo video demonstrates exactly what’s on my mind with the awkward latency.

Reply on news.ycombinator.com