Google, Are You Sure No Country in Africa Starts with a 'K'? | Hacker News Reader

Google, Are You Sure No Country in Africa Starts with a 'K'?

theatlantic.com

2 pointsmasswerk4 comments

> I’ve asked the search engine to name it. “What is an African country beginning with K?” In response, the site has produced a “featured snippet” answer - one of those chunks of text that you can read directly on the results page, without navigating to another website. It begins like so: “While there are 54 recognized countries in Africa, none of them begin with the letter ‘K.’” // This is wrong. The text continues: “The closest is Kenya, which starts with a ‘K’ sound, but is actually spelled with a ‘K’ sound. It’s always interesting to learn new trivia facts like this.”

> Given how nonsensical this response is, you might not be surprised to hear that the snippet was originally written by ChatGPT. But you may be surprised by how it became a featured answer on the internet’s preeminent knowledge base. The search engine is pulling this blurb from a user post on Hacker News ... itself quoting from a website called Emergent Mind, which exists to teach people about AI - including its flaws. At some point, Google’s crawlers scraped the text, and now its algorithm automatically presents the chatbot’s nonsense answer as fact, with a link to the Hacker News discussion

So "intelligent thought", which is "having checked your notions", was implemented as "scraping utterances made" without any expected consistency check.

What may be relevant in this context is the observation that models ultimately converge on their dataset:

> trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point. (…)

> This is a surprising observation! It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else. Everything else is a means to an end in efficiently delivery compute to approximating that dataset.

> Then, when you refer to “Lambda”, “ChatGPT”, “Bard”, or “Claude” then, it’s not the model weights that you are referring to. It’s the dataset.

https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat...

– Is scraping good enough? Can we do without curated datasets? (This would be the most significant encyclopedic effort in human history – and we are by no means ready for this.)

Edit: It may be worth remembering the Mundaneum, https://en.wikipedia.org/wiki/Mundaneum

Seems to be worse than overfitting, because in this realm inconsistency is crucial.

Speaking of feedback loops, I thought it may be appropriate to post this on HN…

Alternative URL: https://archive.ph/7PGMJ

Reply on news.ycombinator.com