I had a number of theories (involving arrays newly allocated, where they should have been reused in a managed stack, and extensive heap allocation, clamped array implementation, side channel attack mitigations running wild, etc). However, as it turns out, it's apparently about hardware acceleration (and probably about some overoptimistic assumptions made).

I.e., I just tried on an old MacPro (late 2008) with hardware acceleration disabled (Settings -> Advanced -> System -> Hardware acceleration + restart) and it runs smoothly.

Edit: Just updated the page accordingly. Thanks to everyone who helped in focusing on the matter of hardware acceleration!