Some examples seemed gimmicky and of questionable value (e.g. dividing the price of gold by the price of silver) - but given the right context, I'm sure it could return useful results for more realistic queries as well.
In one instance, he used the wrong tense ("is" instead of "was") in a query ("How old is Barack Obama in 1967" or something like that), and no results were returned - but the error message wasn't smart enough to suggest why it didn't work.
One big challenge will undoubtedly be to "curate," as they say, enough common data sources to make it useful nearly all of the time...rather than just some, or even most, of the time. I think there's a threshold of usefulness (measured in percentage of successful queries - perhaps 98% or more) that Wolfram will have to reach if it intends not to languish as a mere curiosity like Cuil. Given that they've already got 200 people doing curation of just their existing data sources, and they plan to add many more sources, that seems like a pretty significant financial barrier to cross - I don't know whose pocketbook got the project to this stage, or how they intend to continue before it starts generating income.
If Wolfram Alpha IS a success, it seems like it could change the behavior of web users in two ways:
1. People will start expecting a higher degree of authoritativeness from information they refer to on the web. For instance, if you're making a decision with a committee, and there's an email thread going around that sums up facts upon which the decision will be based, nowadays those facts might include links to "non-expert" sources like Wikipedia. If you have the option of linking to the same information in Wolfram Alpha, you'll probably choose to link to Wolfram instead, since you know the information has been vetted by paid experts. There also appeared to be a "References" link at the bottom of each results page that had an AMA-citation-style list of sources. Although Wikipedia also often links to sources, it's arguably more of a crap shoot as to whether you should trust the source, the editor referring to the source, or even that the reference was made in the right context.
2. People will start using Wolfram as the single (or at least the first) place that they go to for factual, authoritative information. For example, nowadays, if I want authoritative information about a stock, I can go to Yahoo Finance and pull up the financial details - and be reasonably certain that it's accurate and trustworthy. However, if I also need to refer to another type of data - for example, official population data for a region - I'd have to find another authoritative source - perhaps the CIA world factbook. Instead of having to remember multiple sources, it's much easier to go to a single, trusted source for all types of factual data.