> they have some long-context "Memory Model" (whose details I've yet to fully understand)
Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens".
This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models).