First, there are competitors in the space. Some have done an objectively nice job, but they're not blowing up. Second, you don't realize it, but you're not just competing with Google (the search engine) -- you're competing with every product in the Google ecosystem. If you want people to use your new niche search engine, you have to find a value proposition so compelling that people will explicitly choose to use your search, even though every other tool they use is tied back to Google somehow. That's insanely hard.
Finally, but most importantly: programmers are a tiny market, they're terribly picky, conservative, and they don't like to pay for things. If you're going to take on the challenge of beating Google in a vertical, at least make it one that's huge or has huge value. Focusing on programmers means that you'll spend a ton of time trying to optimize for a highly technical, low-value use case that doesn't generalize well.
Solving problems you face daily and are tired of has always been significant driver of innovation. This is an important and satisfying problem to be working on, not all startups have to become the next google or the next unicorn.
No, but you do have to survive. That's hard enough for any company, but it's much harder when you pick a small target market that can't be trivially replicated in other verticals.
So sure, pick a problem that you know well, but be realistic about the achievable market size, and don't start a business that requires huge scale to work (i.e. a search engine), when your target market is tiny.
(Also, as in all things, beware survivorship bias: just because counterexamples to what I'm saying exist, does not mean that the likelihood of creating one is high. Nor does it mean that the counterexamples are doing as well as you think they are.)
i'd like to be able to search google for "all the pages that contain a particular javascript snippet", for example. obviously, google has that data. obviously, they dont want to make it available.
there are services that offer such things but they suck because their datasets are way too small.
on google specifically: the results pages dont matter anymore. nobody clicks anything but the "#1 results" if they click even that. for questions, google very often responds with something like a quora question or some widget of its own, or something else that is really useful - and if its not useful, you try another query instead of clicking through the results pages.
results pages are mostly kept for nostalgia it seems.
that's a good point about only the first few results mattering.
wolframalpha is computer algebra software turned into a web service which allows for searching some papers, pre-written articles and a bunch of interesting, curated numbers.
wolframalpha provides qualitative answers to "questions about math".
answering the physics problem "whats the speed of light" with "er its fast" is qualitative. saying its 2.99+/- epsilon 10^6 km/s is a quantitative result based on data gathered from some experiment.
answering the search engine problem "whats the speed of light" with "2.99+/- epsilon 10^6 km/s" is not a quantitative result. in a search engine contexts, thats just the most likely correct result . qualitative. just a guess. some computer looked at a bunch of data and decided that one of them was probably right based on some popularity heuristic. its like asking an expert who makes an educated guess and claims its right because seniority.
a quantitative search engine would be capable of answering questions like "how many sites do X".
wolframalpha doesnt do that. its just maple in a browser.
https://www.wolframalpha.com/input/?i=IR+spectrum+of+toluene
http://webbook.nist.gov/cgi/cbook.cgi?ID=C108883&Units=SI&Ty...
A search engine that also has a near monopoly in browsers and phones is a pretty big moat.
Since the bulk of browsing is moving to mobile, it's almost like you have to start with an Android killer and worry about the search engine as a second step.