If you got random tokens on Apple Silicon Macs, you should check your stderr for Metal’s error code, like 5 for out of metal memory. Metal lets you use half of your unified memory as metal memory.

Thanks for testing it. I wonder if there’s a reasonable line? 64k? How much RAM did the 128k context require? (Maybe future models might be better at that level)