Hacker News Reader

Comment by photoGrant | Hacker News Reader

The multiverse visualisation was really cool! This whole tool feels like it's designed for a writer from the future.

link for curious: https://generative.ink/meta/block-multiverse/

I have always been bothered with the multinomial layer that picks the next token. It seems like you should be able to stick a classifier in there to detect if the probabilities align with a satisfactory answer and whether the model is unsure of the response. If that is the case, then it should branch out to a bigger model which would compute that single token that it is unsure about, and from that the rest of the response is obvious for the smaller model.

This way you could combine e.g. a 7B parameter model with a 70B parameter model, and get the quality of the larger model while most of the time you are only running the small model.

Edit: You could also store the full probabilities for each token, and then the classifier could detect if it had gone down a bad path, and then unwind the tokens and pick a different path.

That's reminiscent of a caching scheme, you could probably layer that down to even smaller and up to even larger models for increased performance and accuracy.

Better yet: have many smaller models that can when confused call upon larger models and the larger models could then pick the most appropriate smaller 'expert' model or, alternatively themselves escalate. Sort of a supervisor tree for language models.

Sounds like speculative decoding.

https://huggingface.co/blog/vivien/optimal-lossy-variant-of-...

Contextual loading of a bigger model would incur a lot of latency. In this case it might be a good idea to farm out the request to a non-local model. Could be a good avenue for a hybrid local & remote completion system.

I want to say this is where the progress is being made currently. Not only having 'experts per domain' but 'experts per pairing'

Is there anything experimenting with that?

I would be very surprised if (not at all)OpenAI didn't. I've noticed it with both gpt 3.5 and Gpt4 that if you ask it a difficult question the latency increases a lot, especially at the beginning of the answer. I wouldn't ve surprised if they did exactly what was described.

Right I’ve noticed that too. It’s like it takes a pause and reevals. Definitely feels like a cache or memoization of token paths. Interesting.

...from the futures!

Reply on news.ycombinator.com