Hacker News Reader

Isn't that the same transformer at the end of the day? It must be faster only because it generates a single token output, just one evaluation of the model. It takes the same input context and has the same O(n^2) attention blocks. It probably takes options as appended to the input and returns a probability over them instead of the whole dictionary. It's post-trained to do that specific job. If so what's the big deal?

View on HN

They say it's "parallelized". Whatever that means in reality, their demos are pretty good, their prices are extremely low compared to alternatives, and it responds in ~100ms which is pretty fast for what they do. Whether it holds for longer inputs, edge cases, etc. remains to be seen, but I can imagine the use cases for that, for example you can use it directly in the sampling layer of a normal generative model, or just as a generic decision maker/controller. They can (and will, in their words) do this for images too. I don't know if it's a big deal, but it's kind of a fresh perspective.

Unless I misunderstood what they wrote, I read parallelized in the diffusion sense, akin to GemmaDiffusion and Inception Labs models. Incidentally, Mercury 2.5 is truly groundbreaking, giving it a try is highly recommended.

yup https://github.com/vllm-project/vllm/pull/57250

Reply on news.ycombinator.com