> Perhaps aggregation apps should have the client do the scraping, rather than being entirely dependent on server side scraping?
Unfortunately, this is where the RAM Copy Doctrine gets us into trouble. It is not only illegal to "exceed authorized access" to a networked computer, the precedent currently considers loading any copyrighted work into RAM potentially infringing, e.g., if the rightsholder says you're not allowed to use their copyrighted work in that way, you have to present a viable fair use defense.
afaik, no one has brought suit against things like client-side adblockers and browser extensions that modify a page, but if they did, they'd be likely to prevail under current precedent.
We really need true legal protection for users to select their own user agents and to be free to access information willfully transmitted to them in the way they like, especially in the case of something like Facebook/Twitter, where the site itself is just a wrapper around other peoples' copyrighted content.
That will only happen if someone can convince enough Congresscritters to carve out an exception in the actual law, rather than relying on long-outmoded pre-internet judicial interpretations.
Power Ventures scoped down to extract only your own data out of Facebook and they still ended up owing $3M in damages.
See Ticketmaster v RMG at https://en.wikipedia.org/wiki/Ticketmaster,_LLC_v._RMG_Techn.... , where the argument that alternative user agents should be allowed was shot down. I discussed at some length here: https://news.ycombinator.com/item?id=12352450