Jimp is something I had come across as well and played with a bit. FYI I wrote in my notes "dependency on jpeg-js is dog slow." I recommend some perf tests for your use case before jumping too far in.
The decoder.js script in jpeg-js is < 1000 loc, so I suspect this can be addressed if someone comes up with a reproducible case?
Edit: I'll happily try to do it if this isn't resolved by the time I get around to needing to use jimp / jpeg-js.
My personal favourite is node-canvas. It simply brings the HTML5 canvas api to node.
The gory details and a few attempts at higher quality scaling algorithm implementations are on SO: http://stackoverflow.com/questions/18922880/html5-canvas-res.... TL;DR: sending the image to the server and having GM scale it is faster and more reliable than your own scaling algorithm in single-threaded browser JS.