They are right and you are wrong.
If some web page is publicly available, it should be indexed.
Scraping neutrality, please.
You're certainly allowed to try, but I don't see why indexers should be mandated to collaborate with you. They serve their users, not you.
I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.
What if I consider (some or any of) my ideas to be un-indexable, not directly suitable to representation in any hierarchy other than those I may set them in?
example.com/correcthorsebatterystaple
Taking either step you suggest (along with robots.txt or eqiv.), it would seem fair to expect that Brave, Bing, whomever, would not feel it their neutral/natural domain to include in a public index.