JS should not be that slow in my experience. Do you still have the code to your benchmark somewhere? Perhaps you ran into a performance bug that we can file for the relevant browser(s).
I did some serious optimization when I wrote it. There was a "best practices" for V8 that shedded some light on the background of JS variable allocation, I followed it and got a serious boost. Also did lookup tables instead of runtime math with exp/log functions. But in the end: hit several worst case images that took more than 12s, which was a gap that couldn't further be reduced through optimization.
But you're absolutely right, I should redo it in PNaCl.