You should probably report this. Most of the test cases that they used where PDF.js is too slow, involves files so large that even Adobe Reader can't render quickly, or files that are heavily damaged.
I've reported it and uploaded the pdf, so someone on IRC responsible could debug it, but we didn't get far enough to fix it. Otherwise I wouldn't just add another PDF Viewer to a site. But it's only the downloads section and the preview there really is not for reading, but looking if it's the right document. Anyway, thanks for caring for the project. There's a reason Mozilla chose PDF.js over writing it in C++ and that's not just because of security I think.