The problem is that now everyone who wants to build a decent crawler/scraper has to do it as well. Used to be any kid in his parents' basement could put wget and sed together and explore the internet. Now Timmy has to bolt a whole damn browser runtime onto his script to make sure he's getting the content text from blog posts and twitter feeds.

Asking Timmy to "step up his game" is heartless, and it'd be unnecessary if we were better citizens.