The news done broke | Hacker News Reader

The news done broke

b-list.org

14 pointswebology6 comments

Next time they may ask the office to provide those data in PubSubhubbub protocol so that active polling of data is no longer a requirement.

Just a side note:

  All screen-scraping techniques are vulnerable to the 
  “whoops, they redesigned” problem. This screen-scraping 
  technique was vulnerable to the “whoops, they redesigned and 
  we go live in a couple hours” problem.

These two sentences sound a little odd to me. Since the original page by James has no comment box, hopefully he could pick this up here.

Why do these sentences seem odd seem odd?

Pulling data from HTML is highly dependent on its exact structure. Usually if the page changes at all, you have to go back and figure out where everything is again.

Probably by "we go live in a couple hours" he means that they didn't have access to the page and its new format until a few hours before they had to have it ready to send out.

It seems odd because like 90% of the words from the first sentence is repeated by the second one. Seems not DRY enough for me. It's more on the style of text than the problem, as I clearly agreed that by suggesting the data source to adapt to publisher-subscriber model (or some xml/json format)

Well, there are two things.

One is that my first look at the (new) HTML of the page I needed to scrape came only a couple hours before polls closed, thus adding a bit of urgency.

The other is that, stylistically, I'm a fan of anaphora and similar tricks.

There was a period in time where I needed to do the same thing for sports coverage - long story, but there's something amazingly gratifying about ingesting huge quantities of data from semi-public sources in a time-critical way.

> there's something amazingly gratifying about ingesting huge quantities of data from semi-public sources in a time-critical way.

1) the feeling of being completely 'connected'

2) the feeling of being on the 'cutting edge'- knowing before 99% of the population hears about it

Reply on news.ycombinator.com