The typical response to people raising these issues is "buuuut xy is a private platform that can do what it wants", yes, but why are you defending technocrats with bigger profits than many nation state's GDPs? (Reasonable) crawling should be allowed and promoted, in fact, it should be codified in law as a necessary element for the future of open and free internet. Anyone trying to prevent it, or even worse, make it illegal, is a bad actor.
Rate limits can be applied for different reasons. If they protect the website from being overloaded, they are good in our opinion. If they protect it from competition, research or building new non-competitive, but valuable products that are not harmful to the original website, they are not ideal.
We leave that to the user to decide the ethics of their project and just provide the tools.