Hacker News Reader

Comment by throwanem | Hacker News Reader

That's two examples in one day of GPT being bad at scansion.

Q: What are the syllables in the word President?

ChatGPT: The syllables in the word "President" are "pre-si-dent", with stress on the second syllable "si".

So it does know about the nessesary concepts. It can also explain scanision. It just isn't good at putting that into practise. I guess this is one example to show that GPT isn't an AGI, as much as some want it to be.

It doesn't know anything at all, it just puts probable sequences of words together, for some definition of probable.

It knows things in the sense of "you can ask it and it gives you a relevant and factual answer". It can also transfer knowledge to new situations to a degree. But it doesn't know everything, and there are stricter definitions of the word knowing where GPT (almost certainly) knows nothing.

But just because the last layer outputs probabilities for tokens, doesn't mean you can't have intelligence in the layers above that. I don't think "it generates a chain of token pobabilies" is really disqualifying.

The thing that doesn't know anything will still take your job.

Nice. I never really liked working.

While I agree, it will probably also kill us (or at least disrupt our lives in unfathomable ways). Either on the orders of someone else, in the social unrest when unemployment is 70% and basic concepts of human existence are torn to pieces, or when it decides our atoms could be used to increase its reward function value.

Even that isn't correct - the stress in "president" is on the first syllable, not the second.

It's interesting to me that both errors lie in the domain of spoken rather than written language. I wonder whether they'll persist in multimodal models trained on speech, as well as text and images.

Still 100x better than the average person!

I read the Wikipedia article about Scansion and I still don't understand it.

I asked chatgpt to explain the comment:

The comment is referring to GPT's ability to scan and analyze the meter or rhythm of poetry. Scansion is the process of analyzing the poetic meter by marking the stressed and unstressed syllables in each line.

The comment implies that there were two instances where GPT failed to accurately analyze the meter of two different poems in a single day. This may suggest that GPT is not proficient in scanning poetry or that it may not be the best tool for analyzing the meter of poetic works.

tl;dr: when you read Shakespeare plays out loud, there's a rhythm and meter to it. The process for creating that is called scansion.

I'm sure you've measured this.

I'm sure you know that one of the uses of intelligence is the ability to estimate things without measuring them - based on your actual experience and the quality of the model of the world that you have. Thankfully we can use it to know how to e.g. tie our shoes (without having formally studied knots), or to be able to tell intuitively that most people can't understand Quantum Mechanics, even if we haven't measured it.

You are however welcome to do some statistics with actual samples of talking to people and having them show their skills at scansion, and prove my estimation wrong. Heck, I'm willing to bet less than 1 in 100 Americans is able to even define what "iambic pentameter" is. Feel free to prove me wrong on that one too.

Except if your concern is the measurability of the skill: whether it's 100x and not just 10x or maybe just 8.345x better. In which case you've missed the point of using it as a stand-in for "much better".

Just as we can tie our shoes without studying knots, many people can feel iambic pentameter without having any idea what the term means. Many can't, too, or can but only vaguely. Still, those people are still on par with chatgpt when it comes to scansion. I think you'd be hard-pressed to find anybody with a worse sense of meter, let alone 100x worse (again, without knowing any of the terminology etc.)

Reply on news.ycombinator.com