No framework can fully eliminate the underlying performance sinkhole that is the DOM. If you have to mutate, you will pay the price, period. The smartest framework can only help you eliminate redundant mutations.
> I wouldn't even know where to start to allow clicks and such.
Figuring that out should be a good exercise, then. Otherwise, you might want to use some library building on top of canvas.