Because when it first started I think there was a mutual, non-spoken agreement between web content-owners and Google, sort of "I give you our content for free to use it on your search engine and you bring me back visitors by trying to be impartial in your SERPs; but I don't expect you to compete against me, not now, not ever".
That didn't last long, to say the least, but said content owners are now prisoners of Google (yes, it's a big word, but that's the fact). If my personal experience counts, I let Googlebot crawl my website like crazy, at speed rates for which I would certainly ban any other potential search engine bots, and that's because Google brings in 98% of my search engine traffic. And I'm sure I'm not the only one banning strange-looking bots, which could belong to potential new, wonderful search engines, but which at this rate will never see the light of day.
And, unfortunately, things like DuckDuckGo are sub-optimal outside of the English sphere of influence, at least that's what I could see in my case.