Lawsuit nastiness aside, there's an interesting and important legal-technical question that this exposes: how should websites specify acceptable uses of crawled data and other fine-grained restrictions in a machine-readable form.

Motivated by this incident, I got together with Pete (the author/victim) to write a piece on "The Need to Reboot Robots.txt" [1] but it went nowhere.

Any suggestions on how to give our proposal legs would be much appreciated.

[1] http://33bits.org/2010/12/05/web-crawlers-privacy-reboot-rob...