Hacker News Reader

Comment by eznzt | Hacker News Reader

With time everybody learns if your time is worth something you will be scrapping, not using an API.

I dunno, maintaining a scraper across neverending site redesigns is a friggin' lot of work. Way more work than updating my code to use a new version of an API.

When was the last time goodreads was redesigned?

If all that’s changing on a site is the presentation/ordering of elements, then you just need to update some XPath selectors every now and then and have a notification system for knowing when those selectors aren’t working.

You can even automate this process by having a known input (eg, URL to a book on goodreads + its known [and hopefully unchanging] book title) and have a script that periodically checks that the xpath string matches the known pages text / generates a new one to point to the title. This is harder for values that do change but there are always workarounds

I know. But it's like pager duty. When it breaks, I have to manually intervene to fix it, and in the meantime stuff is not working. And anything I build on top of that needs to take that into account.

It turns fun projects into something like a job.

It depends on what you do with the content: if you're just using it for yourself and can afford it to break at random times, scrapping can be fine (although maybe still more time-consuming depending on the website).

But if you need predictability and reliability (e.g. you're providing a service to other people) for whatever you implement using this 3rd party service you don't control, relying on their ui that they can break any time they feel like it will lead to more downtime than APIs for which you're usually given some notice before they're deprecated.

Reply on news.ycombinator.com