Hacker News Reader

Comment by toomuchtodo | Hacker News Reader

If the scraping model is impaired due to aggressive countermeasures, end game are browser extensions that scrape as users view the site and ship scraped data back to a processor, similar to recap the law (uses an extension to scrape the PACER legal database and ship digital artifacts to the Internet Archive). Care will need to be taken around potentially sensitive data that could be shipped if users are logged in.

https://free.law/recap

View on HN

This model also works well for deep web content archiving.

There was a gaming message board where someone wrote a browser extension that would back up all topics someone visited in the background while they were reading them. It became important for archiving as much content from those forums as possible as the forum was in the process of shutting down.

Oh, that's a very cool project! How successful has it been? If it wasn't for Sci-hub that would be a great idea for the scientific publishing world as well.

Very successful. Millions of court filings extracted and indexed, and a crucial component in driving down PACER costs.

https://news.ycombinator.com/item?id=24086570

> I'm the director of Free Law Project. For the case mentioned in the article we actually did a full expert testimony figuring out roughly how much per page it'd cost to run PACER using AWS GovCloud and a handful of other assumptions. It was...half a ten thousandth of a penny per page

https://www.courtlistener.com/docket/4214664/52/15/national-...

https://news.ycombinator.com/item?id=24085158

> Government’s PACER Fees Are Too High, Federal Circuit Says

https://news.bloomberglaw.com/white-collar-and-criminal-law/...

Reply on news.ycombinator.com