After Being Upstaged By Google, Wolfram Alpha Fires Back With A Leaked Screenshot | Hacker News Reader

After Being Upstaged By Google, Wolfram Alpha Fires Back With A Leaked Screenshot

techcrunch.com

23 pointsvaksel25 comments

I think innovations in search will be largely about interface and about relevance. It's not who has the most, or the best data, because it's not really that much of a technical challenge to build a big search index anymore. So, I think that Wolfram could be on the right track, because this seems to try and expose data that Google doesn't, in ways it doesn't.

The challenge is providing useful and relevant snapshots of that data to the user. In this area, Google continues to innovate. Maps, images, video, reddit-like search, your name it... Google is always looking for new ways to reveal good data to people. And Google is definitely good at producing relevant searches for people's phrases and words.

Maybe Wolfram's new search engine will have a good interface, and provide relevant results. I certainly wouldn't put it past them, and I doubt this will be a Cuil flame out. Though I do tend to agree with the sentiment that talking about something before it is released is suspicious. A good search engine sells itself.

its not a technical challenge to build an index of the entire web?

certainly not easy, and not any easier than it was before as the web is much larger

there is also freshness/timeliness which is the most difficult aspect and is becoming more and more important as more data is added to the web faster (one area where googlebot kills yahoo/msn)

Yeah, I'd say it's not much of a technical challenge. Storage is cheap, indexing algorithms are well-known, and the problem space has been heavily explored.

And it's misleading to talk about the entire web. Google may index billions upon billions of pages, but only a small percent actually get shown to the user, or stored in memory. And I would say that a smart kid with a few computers or a couple of bucks can build the index.

Then the hard part starts. Machine learning to correct search placements, based on features like clicks and time in index. Algorithms to detect spam. Anticipation of user data needs. And all the nitty gritty of linguistic analysis to better sort your index.

These types of things require new and innovative thoughts. In my mind, building a search index is just standing on the shoulders of the giants that came before. I'm not saying Google isn't amazing or that building a search index isn't a great homework problem, but the basic part of getting, storing, and serving the data has been engineered and can be engineered again at will.

I'm think you are oversimplifying the problem. Web search is about the long tail you need breadth in your index. It will cost quite a bit more than a couple of bucks to do (storage is not that cheap, nor is cpu), and you'd have to do it repeatedly (and quickly as I mentioned above). And then of course you'll need enough computing power to perform the actual searches and run hadoop on your web index. There are only a handful of serious players in this space (check your site logs) for a reason

In his (her?) defense, he didn't say "search", he said "index". There's a difference!

Building the index is easy(ier nowadys). Making it work for the users is the hard part :)

ha, well op said build a "search index".. its not much of a search index if you can't search it :)

Clarifying what I said, I meant "building a good search engine" is hard, while using an index to make something faster to find is easy(ier).

I agree (with the previous poster) that indexing parts of the web and making it searchable is pretty much a solved technical problem. There are thousands of people who could pull this off. Heck, I probably could if you give me the hardware.

Now doing it cheap, scalable, etc... it's hard. And making it a good experience for the user is pretty much a war on its own. Fighting spam must be pretty hard...

Indexing, in the trivial sense is easy. PageRank (which is old) is already a step beyond (How does one deal with spam? How does one handle redirects, or splash pages that forward you somewhere). Semantic analysis is a step further (what does apple mean?). And then theres everything else (What should happen when I search for "camera", am I shopping, or do I want to find out more about how a camera works?)

In so far as building a search engine involves buying computers, yes, its easy, but then theres everything else.

Any problem is trivializable in this sense. Amazon is just a storefront (except for the recommendations, the user experience, the search, and I'm sure all sorts of domain knowledge that I don't have at a glance)

It's funny how none of you even question what a search engine should be. Everyone gave in to the concept that you want to find web pages.

Wolfram is the first one to say that you need an answer engine. The web has data, you want answers, not necessarily an URL.

When you discuss indexing, you imply keyword matching. That has nothing to do with Wolfram Alpha. If anything else, this discussion was a great way to question conventional wisdom.

I find it hard to believe that keyword matching has nothing to do with Wolfram Alpha.

You are underestimating what is necessary to do implement tractable algorithms on big dataset.

The classic index/search problem would be extrmely simple if it weren't for the big data involved: it is basically hash table lookups plus set intersection, and you're done. Doing the slightest bit more is disproportionally harder.

Improving relevance is probably best viewed as a recommender problem, and just look at the Netflix prize how hard it can be to squeeze the last bits of performance out of their (toy!) dataset.

I will be really impressed when I can ask a question like the following:

"What is the historical cost of solar per watt for the last 30 years?"

and get a meaningful answer, which right now is scattered across various press releases, journal articles, etc.

I don't really care who answers it, google or wolfram alpha.

What unit of measurement is "solar"?

I'm going to guess he asked "how much does solar cost, per watt..."

Sorry, I meant solar power.

This is the first clue that Wolfram is not competing with Google

I know! It's like thinking that Ducati competes with Amtrak. Sure their products have similar objectives, but their function, specs, market, etc are completely different.

Okay, can you provide a bit more detail besides the analogy? For example, reading one of the links referenced in the OP, it says

"It doesn’t simply return documents that (might) contain the answers, like Google does, and it isn’t just a giant database of knowledge, like the Wikipedia. It doesn’t simply parse natural language and then use that to retrieve documents, like Powerset, for example. Instead, Wolfram Alpha actually computes the answers to a wide range of questions — like questions that have factual answers such as “What country is Timbuktu in?” or “How many protons are in a hydrogen atom?” or “What is the average rainfall in Seattle?”

http://www.techcrunch.com/2009/03/08/wolfram-alpha-computes-...

But Google does some of those things indirectly at least. For example, in a Google Spreadsheet I can type

=GoogleLookup("hydrogen", "number of protons")

and the cell returns one.

=GoogleLookup("Timbuktu", "country")

returns "Mali"

and

=GoogleLookup("Seattle", "rainfall")

returns "36 inches".

Now I don't know whether there is some syntax to do these from the main search, but it seems to cover some similar ground. I am guessing since everybody is very excited about Wolfram Alpha that there must be more than this, or qualitatively different.

[Edit following comment below: I am not saying they are competing, I was asking whether the same underlying technology is powering both, for example with Wolfram being a more complex version of whatever GoogleLookup is doing, or whether the algorithmic approach is fundamentally different. I will admit to not watching the 2 hour webcast and looking for a quick answer]

My comment on Wolfram|Alpha limitations:

http://news.ycombinator.com/item?id=584045

Simple example of Google Limitation (I'm sure there are better):

=GoogleLookup("Ruler of France","1821")

=GoogleLookup("1821","Ruler of France")

both return null. What is the entity?

However, if I want to find out when the next AC/DC Concert is, Wolfram|Alpha probably won't be much help.

They are really not competing...Clear?

I wonder how many people actually use Google Lookup instead of just opening a new tab and searching Google

GoogleLookup is useful sometimes. If the rainfall changes in Seattle, your spreadsheet would update...

Although another proton won't likely be discoverd in a Hydrogen atom.

Wow, neat.

GoogleLookup: http://docs.google.com/support/bin/answer.py?hl=en&answe...

Comprehensive function list: http://docs.google.com/support/bin/answer.py?hl=en&answe...

Is today's Alpha webcast archived anywhere?

Now on front page: http://news.ycombinator.com/item?id=584477

They are already failing since they give you a search box that doesn't work. When it finally does work, I probably will be trained not to type into that box.

Reply on news.ycombinator.com