Hacker News Reader

Comment by dougk16 | Hacker News Reader

I'm doing something very similar with my service: https://aytwit.com/thoughter

To the author, there is a simple trick you can pull in order to make the confirmation instantaneous and avoid caching. Have you figured it out? Let me know!

I also use a residential proxy service for all my profile requests, regardless of the identity provider. For some sites like Twitter, Facebook, etc. this is required, and for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future.

Good work!

View on HN

>for something like Hacker News it's simply future-proofing in case they decide to block scrapers at some point in the future

Why are you scraping? We have an API, it's linked at the bottom of every page[0].

0: https://github.com/HackerNews/API

Good question, and thank you for the question. Both at the time of my initial development several years ago, and including now when I just tested it, scraping has faster response times (cache invalidations) to profile updates than the API. This saves the user of my site being frustrated that they put the code in their profile, but they keep smashing the "Check again" button to no avail.

To be fair the API's cache invalidation does seem to have improved since last time I tested it. It only takes up to 5-10 seconds now, but still this is an eternity to an impatient user.

Also since you imply you work for HN, I'll give you my little secret: If you randomly capitalize the letters in the HN username it invalidates the cache. Otherwise scraping the profile would indeed take as long as the API approach. I'm taking a risk with you fixing it now. :)

I hope that helps clarify and I look forward to your thoughts.

I don't work for HN. I meant "we" as in "we the people of HN"

Ah ok good, that means you won't fix the caching..."feature" that I found. :) Thanks for the discussion!

Honestly I need to use this API as well -- on the list of things to do. I left myself some abstraction to have different ways of checking and the easiest was scraping but honestly hitting the API would have been just as easy. Will fix this

Hey thanks for the helpful suggestion!

Reply on news.ycombinator.com