Whilst the publishers are certainly not pulling their weight, I don't think the biggest offenders have per-paper fees. The biggest offender is probably Elsevier, and as far as I know they make their money from selling subscriptions to research institutions and universities.
As someone who reads a fair number of papers, I definitely feel like the publishers' days are numbered, and not just because of Sci-Hub and LibGen. More and more often now, I can find papers that I want, legally, on the first try. Even more encouraging, I find that when I can't find a paper I often just have to wait a few months and try again to find an open Internet source.
Case in point: a few months ago I wanted this paper -- http://bit.ly/2tE4O4K -- but all the CiteSeerX links were broken and apparently had been for a long while. About two months later it's back. This is just one example, but I find this increasingly works with many papers.
There is also ResearchGate, which is a social network to get papers from an authors, which is legal if they are alive. But their sign-up process is also much harder for independent researchers. You have to "prove you are a researcher," as though attempting to get access to research papers weren't prima facie evidence of that.
In CS the worst paywaller is consistently the ACM.
I typically say I'm a "hobbyist researcher" or something of that nature, and will generally have a response in a week, if not quicker. This even sometimes works for papers just published. I always assure the researcher that it's for my personal use only, and won't be uploaded/republished (and I uphold that part rigorously). Sometimes I've been given extra stuff (source code, other data, etc), without asking for it.
So sometimes, you can just ask - and get a copy. It doesn't always work; sometimes you get ignored, or a polite refusal. But sometimes, you get a good response. And I really appreciate that as a hobbyist.
That said, most papers I like to find or obtain are older stuff (the older the better), which in many cases is difficult to impossible to find (I suppose with some travel and physical research I could get them, but that's outside my means and time). But sometimes, a scanned copy can be found, and in those rare cases where the researcher is still alive, I can sometimes get a copy by asking in that manner as well. Sometimes I have contact the last posted research organization (in those cases, I have been informed on occasion of their passing; but I'll usually get a link or something in return for the document).
In a way, it's a "squeaky wheel getting the grease" kind of move; I'm not obnoxious about it - if I don't get a response, I leave it at that. If I get a declined response, I reply with a thank you. Better not to burn any bridges, maybe later I can get a copy.