Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
> there's no easy way to always reliably identify the requester
That's the problem: you can't identify the person to block them in the first place.
Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.
Not sure what your last sentence means.