As you can see with the HTML5 games in this collection, raw JS is quite unpolished.
If you ask me, I think handling all the little things manually is probably best at this point; JavaScript is actually quite a slow language, so frameworks do introduce noteworthy overhead. For example my old computers do NOT like Unity's HTML5 export at all, and current smartphones can be easily bogged down by bad code too. So hand-optimizing it is the way to go at this exact moment.
I think this is why people are investing so heavily in WebAssembly: it'll allow the actual language primitives to be abstracted out in different directions, and provide higher performance to boot. At least if they get the implementation right :P
WebGL is amazing though, I agree; http://acko.net/ is how all websites should work. :D
There are also well-polished JS games like http://zty.pe/ (that don't require WebGL); that game in particular is built on a nonfree JS game engine, but it's a good demo of what's possible with efficient coding.
I've got some similar things I work on for fun but what you've posted is pretty remarkable!
It looks absolutely ridiculous at level 60+. :D
EDIT: Doh, I completely forgot to ask - what litle side things are you working on? I'm curious.
They also have a way to wrap the JS app as a native app similar to Cordova. The games I've tried on my Nexus 5 worked fine with a great user experience.
For Swerve this is a problem because we're aiming for smooth 60fps but in a less twitch-based game this wouldn't be so much of an issue.