Comment by ivanhoe | Hacker News Reader

Comment by ivanhoe | Hacker News Reader

Parent Full thread

Why would it "break for no reason"?! For all I know regExps matching one small piece of the page if far less prone to breaking than parser that has to analyze the whole page. Designer changes one <div> or id/class somewhere in the top of the DOM tree and you can't reach the node that you are looking for anymore. Same goes for regExp of course, but it's looking at a smaller portion of the html, so it's less likely to be affected by small changes in some unrelated part of the page. And any major redesign will break any dedicated scraper, no matter which parser it uses...

Lets try an example. Extract first link address from https://news.ycombinator.com/.

As DOM query:

    document.getElementsByClassName("title")[0].parentElement.getElementsByTagName("a")[1].href

This will break:

* When title element no longer has "title" class.

* When title is no longer a sibling of link.

* When link is no longer 2nd link of its parent.

As regular expression:

    document.documentElement.innerHTML.match('td class="title">.*a href="([^"]*)"')[1]

This will break:

* On any white space change.

* On any new attributes on td or a.

* When ' is used instead of "

* When href includes escaped "

* In most cases when DOM query will break.

Many of those can happen without any server-side changes. It will sometimes works sometimes won't - making it hard to test.

There are cases when regular expression will break less often than DOM but DOM is easier to reason about, more predictable and has less corner cases.

Reply on news.ycombinator.com