468 karma · joined July 14, 2014
My experience and what I’m concerned about is that eventually more time is spent refactoring to enable new features than writing the features themselves. In my experience this becomes worse with time as the models are reluctant to remove behavior, so the code base will grow to support some version of the old behavior together with the new behavior.
So one could pack all of common state, every question, every answer in the same prefill, using attention mask to only let them attend to their logical parent.
Then additionally do position encoding for token based on their logical position rather than physical.
Then the diffusion step also applies an attention mask to prevent bidirectional attention between answers.
> 6. Calculate only 64 possible label scores—not the whole vocabulary.
This I don’t understand though, could you expand this please?
I wonder though if it supports the same claims as Jev: answers are not impacted by other answers to the same questions, nor the existance of other questions? It seems by sharing KV cache all questions will be visible. And I think the diffusion causes the answers to attend to eachother?
Also I think without fine-tuning we can not say that the probabilities it output are actually probabilities. Maybe fine tuning using Brier scoring would do the trick?
Maybe some kind of hierarchical structure of the KV cache can make questions independent, and with smaller diffusion canvas’ generated in batch can be a way to make answers generate independently?
Amazing thing, will replicate.
Saying no will sometimes have consequences, and the counterpart is sometimes unreasonable.
Let’s say we have 10 requests, where 9 of them take 1 second to complete but one that takes 100 seconds. The average time to complete a request is about 10 seconds, but if you experience the requests in series, at any given time you’re much more likely to sit and wait in one of those 100 second requests.
So if you imagine a long series of requests from this distribution and place yourself randomly in the series, the average time to completion is just a bit less than 50 seconds.
This is what is meant by t-weighted, that events with a large t take a larger place.
The transaction in question can remain opaque to the LLM and a %transaction.message% string is resolved in the layer between the LLM and the user.
I understand the point of distorted facts, but what I’m not sure how things are improved by basically having no trust in any facts?
But I have to say that the comparison is not really fair. Comparison is done with a 2 B model vs frontier models that are likely 100s of times larger. Also taalas with their 15000 tok/s inference are suspiciously missing from the comparison.
We need to see the comparison with this framework and useful models, which at present seems to mean ~30 B.
I wonder if it is captivating simply because it syncs cool graphics to audio, like those Winamp visualization filters in the old days.
Feedback: I think it would be beneficial to clarify that there is a trial at the top of the page. Currently we have download and buy buttons next to each other, which I assumed meant that the software can be downloaded first, but will require a key you get from ”Buy”.
The images show Switch 1 controllers. Since the Switch 2 is already out, does this imply that the tracking doesn’t work on Switch 2 controllers?
I don’t like the 3 PC activation thing. It implies that the software needs to call home and will stop working if the company goes under. Or, at least, will not be possible to install anew.
But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for multiple tokens.
The verification step is similar to a prefill with a small batch size. The difference is what we do with the generated logits.
I personally would be wary about fire. Custom electronics without experience, and then putting (assuming here) high energy density batteries in a soft toy handled by little kids. Any accident will be absolutely disastrous. And when you scale up those very low probability failures are bound to happen.
How do you think about this, is this handled already?
Would be interesting to see a comparison to quantized versions of the other models. If this model beats the others also in a fair comparison it gives more credibility to it.
Trivially, with r=0, the error is 0, regardless of how heavily the direction is quantized. Larger r means larger absolute error in the reconstructed vector.
Bandwidth-wise it is worse (more bytes accessed) to generate and do random recall on than the vanilla approach, and significantly worse than a quantized approach. That’s because the reference needs to be accessed.
I guess implied is that since the KV cache is smaller, the probability is higher that the parts it that are needed are in fast memory, and that bandwidth requirements of slow links is reduced, and performance goes up.
Would be interesting with a discussion about benefits/drawbacks of the approach. Ideally backed by data.
> Instead of expecting it to understand my requests, I almost always build tooling first to give us a shared language to discuss the project.
This is probably the key. I’ve found this to be true in general. Building simple tools that the model can use help frame the problem in a very useful way.
One maybe obvious way would be that asking for rumors will actually creates the scenario that the NPC describes.
If, when the speaker actually stops speaking, there is a match vs predicted, the response can be played without any latency.
Seems like an awesome approach! One could imagine doing this prediction for the K most likely threads simultaneously, subject by computer power available, and prune/branch as some threads become inaccurate.