You're proving my point. That's a reference C++ parser that only parses robots.txt files. It doesn't parse meta tags and doesn't help with writing crawlers running in any of the major language ecosystems used by crawler authors.

It's much more difficult than it should be for scrapers who want to be compliant to to actually be compliant.