>for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future
Why are you scraping? We have an API, it's linked at the bottom of every page[0].
Why are you scraping? We have an API, it's linked at the bottom of every page[0].
To be fair the API's cache invalidation does seem to have improved since last time I tested it. It only takes up to 5-10 seconds now, but still this is an eternity to an impatient user.
Also since you imply you work for HN, I'll give you my little secret: If you randomly capitalize the letters in the HN username it invalidates the cache. Otherwise scraping the profile would indeed take as long as the API approach. I'm taking a risk with you fixing it now. :)
I hope that helps clarify and I look forward to your thoughts.