Comment by sigil | Hacker News Reader

Comment by sigil | Hacker News Reader

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype...

Actually, the tech is extremely complex, and PageRank is only a tiny part of that. Try building a search engine sometime. To hackers inspired by this article: here be dragons.

DuckDuckGo took the only sane approach and aggregated results from existing search engine apis, then gradually mixed in some secret sauce. Even then, is DDG viable competition for Google? Will it ever be?

All the same, I join you in wishing for some Innovators Dilemma-style disruption here. A "toy" service comes along one day that's a substitute for Google search, but only within a tiny niche. Google doesn't take it seriously enough until it's too late...and we have a real ballgame on our hands again.

Thought experiment: what could that disruptive niche be?

> Thought experiment: what could that disruptive niche be?

Someone needs to write/buy a search engine and then build a really rich API into the internals of it that lets 3rd parties write customized search engines. How about an auto parts search engine, or a search engine for spanish-speaking people living in southern california? What about a Christian search engine, or a meme search engine?

When someone can empower 3rd party developers to make the same kinds of decisions as Google does, but with different tradeoffs, and they really put the full institution behind supporting that, I think:

A) Google will have a hard time competing because they won't be able to give the personal attention to the needs of what is, effectively, a community of modmakers.

B) This new company will capture the long tail of search. Only a sliver of that is covered right now, with a scattering of niche search engines (Google Scholar, etc).

C) The number of users could be VERY large. It could be the cable to Google's broadcast television.

D) Getting started wouldn't require any massive technological achievements. Just find an underserved niche where even a really really stupid search engine would work better than Google. Write it, figure out how to make money off it it, grow. Start with something that requires only a small index. Slowly expand into additional niches according to what will help keep the company and the tech moving forward.

Google's bet is that any information you can glean from someone coming to a niche search engine can be reasonably approximated with contextual information in the query. That's proved true for many queries, but the key is to find the queries where it's really not.

Sounds like Yahoo Boss: http://developer.yahoo.com/boss/ And that's been around for years already.

Essentially, you get the Yahoo results, and decorate, filter, reorganise, improve, combine as you wish. So you get the organic results, which you can then innovate upon.

And it has survived the transition from Yahoo-powered search results, to Bing powered search results: http://searchengineland.com/yahoo-boss-moves-from-yahoos-ind...

Which means you have API access to the second most-used search engine in the industry. So what better way to voice your discontent with Google by supporting a competitor.

Running a successful search engine is expensive, it needs continuous investment into R&D. That's why Yahoo took a step back and partnered with Bing instead. The level of investment needed just to hold status quo with the existing market runs into billions of dollars a year, something Yahoo baulked at. Microsoft, however, were still strongly inclined to invest that every year.

It's a step in this direction, yes.

But doesn't seem to allow you to do write your own indexing, ranking, and querying algorithms, which is the kind of thing I'm talking about.

Yahoo screwed up so bad here, read the research papers coming out of labs.yahoo.com back then and you realize they were onto knowledgraph before Google and actually had a head start, but they shattered their research division and lost a lot of those search people to microsoft and google. They just didn't have faith to create their own path and also lost their identity by deciding to become an entertainment company.

http://aws.typepad.com/aws/2005/12/alexa_web_searc.html

> Even then, is DDG viable competition for Google?

I'd say "yes". I've switched to DDG as my primary search engine, and I'd say they return a good result about 80-90% of the time. They're definitely not as good as Google, but you can always add "g!" to redirect to Google results.

I'd say their best selling point is their focus on privacy, but oddly they don't seem to be touting it very loudly or trying to make hay from the recent NSA revelations.

When watching talks like http://www.youtube.com/watch?v=vShMxxqtDDs I'd be surprised if Google's search model couldn't be leap-frogged if a competitor were to have live access to Google's dataset.

That if is actually Google's biggest strategic advantage though: the resources to keep a near-live and nicely deduplicated copy of all relevant data on the internet. [PS1]

That's why by far the biggest strategic threat to Google is actually JavaScript UIs + RESTful APIs and the easily accessible data sources they form.

Google's demise will eventually come in the form of the adoption of protocols which allow you to efficiently maintain a live view of a service's public resources.

[PS1] And as per https://news.ycombinator.com/item?id=7011816 click-throughs of course, although I don't see a way to side-step that with Google having implemented [not provided]

> is DDG viable competition for Google? Will it ever be?

For search? Absolutely, it is today.

However, Google figured out a long time ago that they could lock people in by offering a hundred other services too. DDG has no maps, no mail, no image search, no video search, no drive, no docs, no news, no book search, no academic paper search, no patent search, no stock graphs, no language translation, and so on. For some of those, it can link off to other services, none of which are viable competitors to the Google equivalents.

What do you mean by "locking people in"? I don't have gmail or youtube account, or picassa or any of that crap, but just use their search because it is orders of magnitude more relevant than DDG for my queries. And I consider maps, scholar, patents to be part of the search.

> Even then, is DDG viable competition ...

An irritation for me with DDG is them formatting result URLs incorrectly, and then ignoring any feedback about it. Two examples are them adding spurious www. prefixes to Google code results, and leaving out slashes when creating Apple developer URLs.

DDG could distinguish themselves by playing bazaar to Google's cathedral, but they don't appear to. A search engine that uses crowd sourcing and feedback would be disruptive IMO.

The examples. https://duckduckgo.com/?q=apsw - note the second link is on google hosting and shows code.google.com/p/apsw/ but clicking on it gives page not found because it goes to WWW.code.google.com/p/apsw/. https://duckduckgo.com/?q=NSString and note infobox at top which goes to Apple developer but clicking gives an error. The link should have a slash between Reference and NSString at the end.

Crowd sourcing would be really tricky. It's inherently easier to manipulate, the more traction you got the more resources would be spent on gaming the crowd sourcing system. Look at what those "reputation management" firms are doing these days with sophisticated blackhat SEO and wikipedia astroturfing rings for example.

Some kind of bitcoin-esque system where you bought sponsored positions by doing search engine scoring related number crunching could be interesting though. Since, if you ever matter, you'll have people devoting resources to trying to game your search engine results you need some system to deal with that. Of course it's sooo "out of the box" to suggest a bitcoin inspired solution to things right now ...

Scour (http://scour.com) has been trying to make a go of "social search" for a while.

Vertical search engine might be the answer, much easier to implement and much more useful from users point of view.

I think vertical search engines are part of the answer too. For general search engines Google have such an advantage in data and R&D that only Microsoft can afford to compete.

On the data side I think http://commoncrawl.org/ can help with creating vertical search engines. Their crawl is much smaller than Google or Bing but it is web scale (2 billion pages of 2013 data). Data recency is still a problem but it can help with finding which sites belong to a niche. Some smaller scale crawling of these sites would then be a much more achievable task.

> Thought experiment: what could that disruptive niche be?

Internal search engines could get a lot better. I don't know how people who make websites make them, what tech they use etc., if it's mostly incompetence that does this, but the fact that I'd rather use plain Google than an internal search most of the time tells me that Google is just too good.

Reply on news.ycombinator.com