I think vertical search engines are part of the answer too. For general search engines Google have such an advantage in data and R&D that only Microsoft can afford to compete.

On the data side I think http://commoncrawl.org/ can help with creating vertical search engines. Their crawl is much smaller than Google or Bing but it is web scale (2 billion pages of 2013 data). Data recency is still a problem but it can help with finding which sites belong to a niche. Some smaller scale crawling of these sites would then be a much more achievable task.