If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google.
I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site.
I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?)
So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.
There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)
This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.
And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.