Yes, this. The whole purpose of robots.txt is to allow restrictions to be applied on objects that the crawler hasn't looked at yet. A META tag can only apply restrictions after the fact.

Moreover, his proposed replacement is incredibly browser-centric. It requires any compliant crawler to contain an HTML parser. And woe befall anyone who typos the META tag, or manages to confuse the HTML parser before it gets there! The robots.txt specification, by contrast, requires no such heavy lifting: just request a single fixed URL.

META tags make perfect sense for favicon and the like - I won't dispute that. Robots exclusion, however, is a special case - it belongs outside HTML, not inside it.

Just for clarrification: 1. robots.txt excludes CRAWLING e.g. downloading but NOT indexing, e.g. including a URL / site in a database of known URLs / sites. 2. Robots meta tag disallows INDEXING but NOT crawling.

So it is semantically correct, although most modern SEs do not do this, to index a site / URL that is disallowed via robots.txt, using link data alone.