1. It's really cool that you guys are making your own models. It was looking like OpenAI might end up with a Google-style dominance of these things but apparently it's still possible to train one and get good results without them!
2. It's a bit unclear how programming specific it is. The sample queries are very programming dominated. Every results page seems to include a code snippets section even for non-programming queries like your Elon Musk query which presents a couple of "code snippets" that's just meaningless garbage. Presumably because it's not a coding question.
3. I tried a real question I had earlier today whilst programming, and which I'd also tried on ChatGPT. The first paragraph is the correct answer!
4. It seems to have trouble disambiguating different languages. My question was how to do something with the AWS S3 Java API, but the cited links were of a JavaScript repository showing how to solve the task. It also tried to generate a code snippet which was just totally garbled, it was obviously meant to be JavaScript but it wasn't even wrong, just non-syntactical nonsense.
5. In the same vein it showed me a code snippet that claimed to be Java but which was clearly C# (or meant to be C#).
It did manage to cite the AWS SDK docs but only right at the bottom. That was the most useful results. For comparison when I asked ChatGPT a very similar question earlier, it hallucinated a completely convincing answer which relied on an SDK method that didn't actually exist. Sadly not so useful. I think I marginally preferred the Hello results.
6. I gave it a really hard test by asking who I am. It's tough because there are several guys with my name who have a web presence. It proceeded to mash together details of all our lives into a single biography. The citation feature was useful here because it made it clear that several different people were being conflated, whereas the answer alone might have sounded convincing. On the other hand, just doing a regular web search and picking a page about the right person would have been faster and clearer, without risk of being misled.
The big questions in my mind are:
1. Business model? As someone with a developer tool to sell I've been kind of frustrated at how useless AdWords is for communicating the existence of tools to developers. They could surely use some competition there.
2. Truthfulness still seems like a fundamental challenge. The citation feature makes it really much more obvious but is that really a win? I finished by asking it a straightforward question about the population of my local city that Google answers instantly in the auto-complete box, I don't even have to press enter. Hello gave an answer that's roughly right but the snippet of the citation made it clear that the actual answer was a different number. The number the AI generated doesn't seem to appear anywhere in the cited page.
Overall, really exciting to see a LLM that can both cite its sources and is exposed as a product! But right now I'm not sure it's faster to use this than regular web search :(