It ticks me off that bots no longer respect robots.txt files at all. The authors of these things are complete assholes. If you’re one of them, gfy.
Giving someone a "financial incentive" to do something (by gasp using a search engine, or comparing prices) does not make that thing ethical or cool to do in and of itself.
I wonder where you ever got the idea that it does.
You're shifting the frame, to one where morality doesn't come into it. You're asserting some kind of market inevitability, which is probably the same sort of rationalization arms and people traffickers use to sleep at night.