> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired.

I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance.

Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a new URL, here's what you do:

- Go back to one of the old URLs where you already have the correct value (let's say it's the title).

- Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo <span>Bar</span>" which you want to match against "Foo Bar". So this is going to be an expensive search.

- Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc).

- Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector.

Any thoughts on whether something like this would work?