The daftness: maybe the claim is true that robots.txt was only a stop-gap measure back when web servers sucked, however the de facto modern use for it goes far beyond that, and ignoring that standard is likely to piss off lots of people.
The insight: for crawlers, relying on robots.txt to prevent getting stuck indexing infinite hierarchies of data is a bad idea. It should be able to figure that much out for itself, so it doesn't explode when faced with sites that don't exclude such hierarchies using robots.txt.
For servers, relying on a client hint to ensure reliability is daft. It should have some form of rate limiting built in, as that's the only sensible design. This seems the only marginally sensible use of robots.txt from a server standpoint. Using it for any form of security (e.g. preventing DB scraping) is daft, and a more robust mechanism should be employed there too.