Speculative decoding is software, though.
How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
How do you work around the memory wall when you're going to have to stream all weights, no matter what? Latency-hiding tricks don't matter when you're bandwidth constrained.
No comments yet.