Yogi Berra wisdom for startups
redeye.firstround.com
redeye.firstround.com
However, the situation today is quite unlike the day when Google was born. Back then, most search results were almost a collection of random links with keywords on them. Then Google found a fairly simple new approach, Web-link analysis (PageRank), that totally outperforms the keyword-based approach. And Google was the first to exploit this approach on a large scale. (Yes, there were other researchers who discovered a similar approach based on Web links and also outperformed all those other large search engines at the time. But they didn't commercialize the work well enough.)
Now, there is little chance that such a low-hanging fruit is still left undiscovered. To improve search further, it's likely that you need to either or both of the followings:
1) bring in additional information, like user personal preferences. (Google is doing this with their personalized search.)
2) deal with very complex issues like natural language processing (e.g. reading the Web like humans do), semantic Web, inference with ambiguous chain of evidence, other advanced AI issues. (Google probably bought the whole of Powerset just to help make this problem a little more tractable.) (Chacha.com sidesteps the issue by using real humans to do the work.)
I'd say unless you're a real genius and has been studying NLP & AI for a long time, it's extremely unlikely you'd beat all the AI experts at Google. Since AI & NLP are inherently complex stuff, no single paper or invention is likely to break a total new ground as in Physics (think Einstein's Special Relativity).
Sorry to disappoint, but there are certain product areas where likely improvements will be just incremental. We'd better choose our terrain well. Justin.TV didn't compete directly with Youtube for this reason.
That's why, as a startup, you don't really want to compete with Gillette in shavers or Intel in PC microprocessors (or Hacker News in news for hackers, for that matter) by out-inventing them. We'd have a much better chance by finding a whole new ball game to play.
All you need is better search. Is that possible? Google would like you to leave that research to them, with all their money, experience and PhD's, what's the point in trying? Surely there are better prospects anywhere but there? This is one of Google's competitive advantages. (of course, if you did invent better search, they would immediately buy you).
Since Google launched, the Internet has grown considerably. Not only in content, but the types of content that get indexed.
I think Google hit that sweet-spot when they launched -- the Internet was big enough that a good index was really, really needed, but it wasn't too big for them to get into the game.
So yeah. It's possible, but a generalist search engine would be difficult in my view -- Of course, there are loads of opportunities for specialist search engines. But I think these are less of a threat to Google (and you're right, Google would probably just buy them up).
But research, pre-startup, might reveal better search.
However, thinking in terms of Innovator's Dilemma, most search technologies would be "sustaining" technologies, which Google as a well-managed leader could buy, copy, re-discover or invent around and keep their lead. It's so-called "disruptive" technologies that would cause them problems (which aren't dangerous because they're better, but because they're different in their benefits) - but these are a problem for any company. So Google's position is no more tenuous than any other company's.
This is true for one user. But BILLIONS of people are using Google. And it's NOT easy for all of them to switch. Google's position is not tenuous at all.
Google relies on a massive network effect to hold their position - this makes them a very stable system in many regards. However, there is also the risk of a collapse scenario.
In the case of Google, it might only be one or two segments (in whatever demographic/dimension) that are key.
It would take some time for billions to switch, yes; but if there was a search engine that was far superior for what users need, the difficulty of switching would not prevent them. It's the standard pattern of technology adoption.
People get attached to consumer goods (e.g. brands of washing powder, chewing gum), but while they also get attached to technology, it doesn't seem to do the technology much good once it has been superseded (DEC anyone?)
The lack of a "far superior search engine" is the significant barrier, IMHO. And whether such a technology is possible is something I don't claim to know. That's the unpredictable nature of technology/science, and why people like Warren Buffett won't invest in it.
Great post.