Hacker News Reader

Recent proofs of concept like GPT-3, though far from perfect, make me optimistic that good Q&A models are within reach.

Q: Did Finland join NATO or the Warsaw Pact?

A: Finland was not a member of either NATO or the Warsaw Pact.

Q: What is the most populous city north of the Arctic Circle?

A: Barrow, Alaska

Q: Does GPS accuracy depend on special relativity or general relativity?

A: GPS accuracy depends on general relativity.

Q: How may Boeing and Airbus planes have been built?

A: Boeing has delivered over 10,500 aircraft, while Airbus has delivered over 6,300 aircraft.

Q: What is an example of an undecidable question?

A: The Post correspondence problem is an example of an undecidable question.

Q: Is suicide ever the right thing to do?

A: Suicide is never the right thing to do.

Q: Who broke up the Beatles?

A: John Lennon, Paul McCartney, George Harrison and Ringo Starr

Q: Who has beaten Tom Brady in the Super Bowl?

A: Eli Manning and Nick Foles have both beaten Tom Brady in the Super Bowl.

Already not too shabby.

Thank you so much! Andi's strong point is these type of factual answers, and I think you're 100% right to be optimist about models being within reach.

The question answering feature has only just been released for the first time today for this post, and for an entire field of questions it already surprises us that not only does it work but it does *well*. We can iterate on the intent error correction and verbal tricks. And we're just a tiny team standing on the shoulders of giants. The entire field is moving quickly and making astonishing progress.

The exciting thing in this area is the rate of improvement. The thing language models have lacked is factual accuracy, and that's definitely a hard challenge. We have problems to solve with applying common sense and reason to things like information safety/confidence, and fixing misunderstood intents is mostly just iterative training. But the exciting thing is that this already works in many cases.

It's interesting to try it out with current news too. Something from today like "why does tesla want to split its stock?"

You can see the progress in this space is real and getting faster. The verbal tricks are fun to laugh at, but the underlying progress is real.

A story about GPT3 is on HN now

https://news.ycombinator.com/item?id=30832465

Interesting article because it talks about all the things that happen with the use of large language models on their own. Large language models are amazing at mimicry and composition, and are a key part of getting to great Q&A.

But on their own they have no idea of factual correctness. That's what excites me about what we're doing with Andi. The answers are not only well generated, but do well on factual questions, especially given this is the first day live. There are some non-GPT models we're using that do well at this too.

Are you doing much with language models at Kagi yet? It's a fun area to work on.

Thanks for trying out all these questions on Andi and posting the results here. That was really exciting to see!!

If you have access to any of the GPT-based Playgrounds, you'll see that large language models on their own tend not to be good on factual accuracy. At the same time, we couldn't build Andi unless we were standing on the shoulders of the amazing work done by the folks working on those, especially the pioneering work done by OpenAI, which has also created an entire open source ecosystem around GPT-J/NeoX etc.

Thanks, I tried:

Q: "who is the current president" A: The current president of the United States is Joe Biden since January 20, 2021

Q: "Who is the previous president?" A: Pulls a snippet from the top of the wikipedia article 'List of presidents of the United States'

Q: Is this a leap year A: Got the same response about Sci - Fi LOL thank you, added to the bugs list! He's a friendly bot :)

Q: When did Neil Armstrong land on Mars A: Pulled a snippet of text from Neil Armstrong's wiki page and says I found this information on wikipedia.org:

I tried a different query,

Q: did Neil Armstrong land on the Moon? A: Yes, Neil Armstrong landed on the moon.

It's experimental, but we know it can only improve over time and we're really excited people are trying out! And I agree that we need to add additional parameters for when do preform a deep answer vs. when to just display the results.

Thanks very much for trying it out and for giving us feedback. We can't see what people search, so it's really helpful to see reports when things go wrong.

For "who is the current president" I get "The current president of the United States is Joe Biden since January 20, 2021". Did you see any results at all? I'm guessing something glitched there. The previous president I just got the same as you, so that's one re-training!

It's early days, but it tends to do best with factual extraction with concrete and specific terms included. Linguistic references (anaphors, previous, next) are still works in progress. But it's heading in the right direction. So you can join concrete questions with more complex phrases and it has enough to work with.

So if you ask:

"who is the 45th president of the the USA and where was he born?"

you get:

"Donald Trump is the 45th president of the United States and he was born in Queens, New York City."

The leap year one was way off! The intent detection was way off the mark there!

Thank you again for giving is feedback. It really does help having examples because we can't see what folks search. The encouraging thing there is that it is almost always misunderstanding the question when things go wrong, rather than identifying inaccurate facts (so answering the wrong question correctly). You can see that with these examples.

Our Discord community likes to have challenges to break Andi's answers, and it really helps us see where it is strong and where it needs the most work. There's a whole school of thought around "tricking AI" and we know how to trick Andi pretty well. But one of the advantages of the search approach is that the underlying results are always displayed alongside the answer. As with many early products, having people use it and give us feedback on what goes right and wrong is incredibly valuable for us, as we can keep iterating and improving the things that go wrong.

So we're grateful for every example of where it is off the mark as well as does well.

I didn't realize when I first replied that you're Vlad who runs Kagi.

Thanks for taking the time to try out the new Andi Q&A features and give us feedback on some of the trickier types of NLP queries.

As someone running an alternate search engine who has domain expertise, that explains your example queries. They are all good examples of the types of natural language queries well-known as edge cases for mis-triggering nlp intent detection. Because we don't log searches, it's really helpful for people to try out these sorts of edge cases and let us know where Andi can do better, so I'm grateful for the comment.

It raises a really interesting point that I missed before though. So I thought I'd post a quick follow-up.

Folks have learned to search Google over the last 20 years using Google's language - keyword searches. That's because otherwise it gave them bad results. So people speak "googlese".

With recent advances in NLP, it's possible to better understand natural language, and this will keep getting better. Natural language queries have some huge advantages when you're searching. Humans are smart at communicating subtle signals and clues to each other through language, and we can use this talent to get better search results.

"Tricky" nlp questions are mostly about using missing context or linguistic implication to fool a system. But as when you're talking with a stranger, as humans we can get around that by providing the information explicitly. When you combine that with language models and semantic search techniques, the quality of the results can go way up.

Part of building a new search application like Andi is teaching users how they can get better results by asking questions with more specific information and "clues".

Unlike Google, with Andi you can get way better results by asking questions with more specific information. Your "tricky" example nlp queries actually make for a great case study in this.

So it's worth looking at how someone can ask those questions in a way that a search engine using language and language models would get the best results.

For the first one "who is the current president" Andi already gets this right, so I'm guessing it was just timeout or glitch and it fell back to search results.

Let's step through examples of how to ask those questions well, and the answers Andi gives:

Q: who was the most recent president of the united states before biden?

A: I found this deep answer on wikipedia.org: Donald Trump

Q: is 2022 a leap year?

A: No. 2022, is not a leap year. Source: Wolfram|Alpha

People learn how to use tools to get done what they need to, and google pidgin is an example of that.

As people discover that they can save time and get better results with a new tool where you can ask more naturally, we think they will adapt quickly.

Thanks again for the chance to talk through this. Your question was a great lead in to the topic.

"Who was the previous president" or "is this a leap year" are hardly questions designed to fool an NLP system, but very common, real queries you can expect from users. Google has no problem handling those.

Actual question designed to fool NLP systems is "When did Neil Armstrong land on the Mars?" which your system did correctly handle, but Google doesn't. Hope that helps.

Thanks. There are definitely edge cases from a technical perspective that are common in the wild.

As the models get better these will get easier to handle, but we think that we can help guide users to asking more specific questions in order to get better results. That's especially the case with spoken search. And we have some ideas on how to help with that which we're working on but aren't fully evolved enough to share yet.