PHP class to query the web by an SQL like language

jonasjohn.de

11 points·alifaziz··10 comments

If only there was some way to address XML/HTML elements by their location in the DOM tree. The "path", if you will. We could call it... XPath.

I'll bet you a nickle that most of the pages you'd use this on are not properly formed XML.

Not as big a problem as you might think. Just feed them to a parser designed to deal with broken HMTL, and then use XPath. I'm using lxml in Python for this, and it's working very well for me.

Likewise on .NET, the HTML Agility Pack: http://htmlagilitypack.codeplex.com/

Not a big deal. You can make a DOM from anything.

In fact, the invalid case is exactly why you'd use this technique. If you use a bunch of regular expressions, you can probably make something that works on a subset of valid inputs. If you want it to work on everything, then you need to use a parser to generate a DOM, and then interact with the DOM.

Or even CSS. Nokogiri lets you use both, even on badly formed pages.

More like querying HTML, not "the web". I initally thought this was Google search on steroids (which would be awesome) but it wasn't.

I've used this library in the past, but it is really too bad it hasn't been updated since 2006. I've since moved on to PHP's DOMDocument::loadHTML function.

another powerful alternative is http://simplehtmldom.sourceforge.net/ this allows jquery selector syntax to be used, which is more likely to be in the skill set of most web developers, and sql might not be.

i've used this and i like it. but it leaks like a sinking ship.

Reply on news.ycombinator.com