"Each of these robots takes up a considerable amount of my resources. For June, the Googlebot ate up 4.9 gigabytes of bandwidth, Yahoo used 4.8 gigabytes, while an unknown robot used 11.27 gigabytes of bandwidth. Together, they used up 45% of my bandwidth just to create an index of my site."

I don't suppose anyone has considered making an entry in robots.txt that says either:

last change was : <parsable date>

Or a URL list of the form

<relative_url> : <last change date>

There are a relatively small number of robots (a few 10's perhaps) which crawl your web site, all of the legit ones provide contact information either in the referrer header or on their web site. If you let them know you had adopted this approach then they could very efficiently not crawl your site.

That solves two problems;

@ web sites on the back end of ADSL lines but don't change often wouldn't have their bandwidth chewed by robots,

@ The search index would be up to date so if someone who needed to find you hit that search engine they would still find you.