Comment by andrewljohnson | Hacker News Reader

Comment by andrewljohnson | Hacker News Reader

I think innovations in search will be largely about interface and about relevance. It's not who has the most, or the best data, because it's not really that much of a technical challenge to build a big search index anymore. So, I think that Wolfram could be on the right track, because this seems to try and expose data that Google doesn't, in ways it doesn't.

The challenge is providing useful and relevant snapshots of that data to the user. In this area, Google continues to innovate. Maps, images, video, reddit-like search, your name it... Google is always looking for new ways to reveal good data to people. And Google is definitely good at producing relevant searches for people's phrases and words.

Maybe Wolfram's new search engine will have a good interface, and provide relevant results. I certainly wouldn't put it past them, and I doubt this will be a Cuil flame out. Though I do tend to agree with the sentiment that talking about something before it is released is suspicious. A good search engine sells itself.

It's funny how none of you even question what a search engine should be. Everyone gave in to the concept that you want to find web pages.

Wolfram is the first one to say that you need an answer engine. The web has data, you want answers, not necessarily an URL.

When you discuss indexing, you imply keyword matching. That has nothing to do with Wolfram Alpha. If anything else, this discussion was a great way to question conventional wisdom.

I find it hard to believe that keyword matching has nothing to do with Wolfram Alpha.

You are underestimating what is necessary to do implement tractable algorithms on big dataset.

The classic index/search problem would be extrmely simple if it weren't for the big data involved: it is basically hash table lookups plus set intersection, and you're done. Doing the slightest bit more is disproportionally harder.

Improving relevance is probably best viewed as a recommender problem, and just look at the Netflix prize how hard it can be to squeeze the last bits of performance out of their (toy!) dataset.

its not a technical challenge to build an index of the entire web?

certainly not easy, and not any easier than it was before as the web is much larger

there is also freshness/timeliness which is the most difficult aspect and is becoming more and more important as more data is added to the web faster (one area where googlebot kills yahoo/msn)

Yeah, I'd say it's not much of a technical challenge. Storage is cheap, indexing algorithms are well-known, and the problem space has been heavily explored.

And it's misleading to talk about the entire web. Google may index billions upon billions of pages, but only a small percent actually get shown to the user, or stored in memory. And I would say that a smart kid with a few computers or a couple of bucks can build the index.

Then the hard part starts. Machine learning to correct search placements, based on features like clicks and time in index. Algorithms to detect spam. Anticipation of user data needs. And all the nitty gritty of linguistic analysis to better sort your index.

These types of things require new and innovative thoughts. In my mind, building a search index is just standing on the shoulders of the giants that came before. I'm not saying Google isn't amazing or that building a search index isn't a great homework problem, but the basic part of getting, storing, and serving the data has been engineered and can be engineered again at will.

I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple of bucks to do (storage is not that cheap, nor is cpu), and you'd have to do it repeatedly (and quickly as I mentioned above). And then of course you'll need enough computing power to perform the actual searches and run hadoop on your web index. There are only a handful of serious players in this space (check your site logs) for a reason

In his (her?) defense, he didn't say "search", he said "index". There's a difference!

Building the index is easy(ier nowadys). Making it work for the users is the hard part :)

ha, well op said build a "search index".. its not much of a search index if you can't search it :)

Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier).

I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware.

Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...

Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?)

In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else.

Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)

Reply on news.ycombinator.com