There's something extremely wrong with your implementation as it just takes too much to render every page.
I've done plenty of work in the past with both canvas and pdf.js (which is what you're using) and it shouldn't be that slow, at all. Perhaps you have a rogue loop that's calling a very expensive function on each pixel of every page, maybe?
Who knows, but for sure performance on that could be near real-time.