Please, Yahoo, distribute the final public-facing database for free. There are millions of links organized in there and it's an incredibly useful repository. If that data is lost, we just lost thousands of man years of tagging the Web.
I don't remember the robots.txt rules for sure, but doesn't that mean they don't allow crawlers at all?
You can't write a simple scraper that is not distributed in 100 of machines across the web to pull out their data.
Actually I just noticed you get 750h of free micro instance time from aws... I wonder if it would be worth doing. I imagine the link+tags are <100GB in total.
That is the most insightful thing I've read today.