I would absolutely expect any drawing application to drop to 0% when I'm not interacting with it. That's the norm with native applications, it shouldn't be any different for a web app.
I'm sure there's more improvements I could make on performance, maybe I could switch to only drawing the mouse cursor on the mousemove event rather than using the animation loop, but then when you're tweaking values in the menus, you won't see your changes in real time any more. It's all a balance, and this is just something I do in my spare time, I'd rather have fun building some new features than seeing my cpu usage drop by a percentage point.
For another data point, on my half decent desktop, it uses about 10% cpu and 20% gpu