I ended up writing asm for the important part:

https://github.com/cloudflare/jpegtran/blob/vlad/arm/jchuff_...