Hacker News Reader

Comment by ggambetta | Hacker News Reader

I've written scrapers over the years, mostly for fun, and I've followed a different approach. Re. "don't interrupt the scrape", whenever URLs are stable, I keep a local cache of downloaded pages, and have a bit of logic that checks the cache first when retrieving an URL. This way you can restart the scrape at any time and most accesses will not hit the network until the point where the previous run was interrupted.

This also helps with the "grab more than you think you need" part - just grab the whole page! If you later realize you needed to extract more than you thought, you have everything in the local cache, ready to be processed again.

View on HN

You're not the first person in this thread to suggest grabbing the whole page text. I've never tried, just because I assumed it was so much space as to be impractical, but I don't see the harm in trying!

My current favorite cache for whole pages is a single sqlite file with the page source stored with brotli compression. Additional columns for any metadata you might need (URL, scraping sessionid, age). The resulting file is big (but brotli for this is even better than zstd), and having a single file is very convenient.

It is not advised to use SQLite with several threads, are you running only a single thread?

You can't use SQLite from multiple threads if you are in single thread mode [0]. If you are in multi-thread mode you can use multiple connections to the DB from different threads, or if you want to share a connection then you have to serialize over it. Or you can use SERIALIZED mode.

This is if you do not care writing any logic. If you do, there are other possibilities, for example using a queue and a a dedicated thread for handling database access, but I personally do not think there are many advantages to this more complicated approach.

[0] https://www.sqlite.org/threadsafe.html

Thanks, I wasn't aware of this. I'm not sure if it was old info or I "imagined" it but in any case it's good to know.

I hope it wasn't true in 2019, since I painstakingly wrote a multi-threaded app that was accessing SQLite on only 1 thread!

I would use SQLite between each of the stages so each process can be restarted, run concurrently and observed with queries.

But how do you deal with dynamic pages, i.e. content that changes each time - would you need to pattern-match?

Reply on news.ycombinator.com