Our project was a bit odd, in that we had millions of documents available to browse. We wanted them all to be available to search engines, but it was an operational headache to keep that much data moving.
Our normal daily traffic might have been 10,000 users with perhaps 50K page requests. We built our first small web server to handle this load.
We were caught blindsided by the Googlebot, which from day one demanded hundreds of thousands of pages per day. Googlebot offers no option to slow it down, and if you try to slow it down on purpose your search ranking will suffer.
So 90% of our early operational burden was devoted to keeping Googlebot happy. This was an unexpected burden, but it was worth it to be in the Google index, because they sent a lot of traffic our way and kept our fledging startup alive.
But it was definitely not worth it to spend equivalent operational expense to be in marginal search engines that don't refer much traffic. To a site operator, they don't offer any value in exchange for what they demand.
It was very interesting to see the relative volume of requests that we got from Googlebot versus the other crawlers. While Googlebot found us on day one, it took over 3 months before we appeared in Microsoft's index, and even then, they had perhaps only 10% of the coverage of our site when compared to Google.