If that is so why does the indexer bother fetching these files at all? It's a waste of bandwidth and cycles to be doing this extra fetch of robots.txt for countless sites.
Kidding me? The average size of robots.txt is 500 bytes for one web site! Average web page size is 2Mb. So if a crawler respectfully downloads a robots.txt it could significantly save more bandwidth not downloading protected pages.