SVGs currently don't support hardware acceleration, so that's why I went with CSS transforms for other parts of the UI. The only things rendered with SVG at the moment are the markup, the UI icons, and the rotation control.
I'm hooked on SVG's promise of semantic graphics, but every time I tried building something with SVG in a browser the performance just wasn't there, especially on low-end devices.
Filters of course work best in webgl. But the rest?
Also, do you use offscreencanvas in chrome?
I currently use WebGL for display preview and 2D canvas for generating the actual image.
I suspect you'll run into performance issues if you plan to animate the image in a 2D canvas context (like I do when rotating / flipping / etc.)
I've been planning to use offscreencanvas but haven't gotten around to it yet.