As a scientist, pdf.js is simply not capable enough for any serious work with pdf.
I would love to do as you say, and read pdfs in the browser only. From what I have learned in the context of this discussion, switching away from pdf.js will contribute to enabling that.