I'd love a great solution to this problem and I'd even consider trying to build one, but I am not sure there is any money in it.
One thing I'd love in an academic paper reader is something that allowed comments/annotations inline with the paper. For example, if a paper in the future contradicts something that is stated, you could add a comment linking to the contradiction. Or you could merely ask and provide clarifications, or comment on simpler alternatives to given part of the paper.
Also, I'd like to be able to rate papers for say, readability or difficulty, tag them as theoretical or empirical, etc.
I'd also like if cited papers were automatically dereferenced so I didn't have to hunt down the references myself.
Personalization would also be a nice feature. EG, recommend other papers by the same author, or other highly cited papers that cite/are cited a given paper, or frequent co-authors of some author that I like.
I'd love to be able to download a bunch of papers easily for offline viewing.