Hacker News Reader

Comment by AndrewKemendo | Hacker News Reader

Can someone tell me where your average every day human that’s walking around and has a regular job and kids and a mortgage would land on this leaderboard? That’s who we should be comparing against.

The fact that the only formal comparisons for AI systems that are ever done are explicitly based on the highest performing narrowly focused humans, tells me how unprepared society is for what’s happening.

Appreciate that: at the point in which there is unambiguous demonstration of superhuman level performance across all human tasks by a machine, (and make no mistake, that *is the bar that this blog post and every other post about AI sets*) it’s completely over for the human race; unless someone figures out an entirely new economic system.

View on HN

The average person is bad at literally almost everything.

If I want something done, I'll seek out someone with a skill set that matches the problem.

I don't want AI to be as good as an average person. I want AI to be better than the person I would go to for help. A person can talk with me, understand where I've misunderstood my own problem, can point out faulty assumptions, and may even tell me that the problem isn't even a problem that needs solving. A person can suggest a variety of options and let me decide what trade-offs I want to make.

If I don't trust the AI to do that, then I'm not sure why I'd use it for anything other than things that don't need to be done at all, unless I can justify the chance that maybe it'll be done right, and I can afford the time lost getting it done right without the AI afterwards.

"The average person is bad at literally almost everything."

Wow... that's quite a generalization. And not my experience at all.

The average person can’t play 99% of all musical instruments, speak 99% of all languages, do 99% of surgeries, recite 99% of all poems from memory etc.

We don’t ask the average person to do most things, either finding a specialist or providing training beforehand.

One cannot be bad at the things one doesn't even do. None of this demonstrates that humans are bad at "literally almost everything."

> One cannot be bad at the things one doesn't even do.

??? If you don’t know how to do something you’re really bad at it. I’m not sure what that sentence is even trying to convey.

> Obviously you could train someone to recite the The Raven from memory, but they can’t do it now.

That doesn't make them bad at reciting The Raven from memory. Being trained to recite The Raven from memory and still being unable to do so would be a proper application of the term. There is an obvious difference between the two states of being and conflating them is specious.

If you want to take seriously the premise that humans are bad at almost everything because most humans haven't been trained at doing almost everything humans can do, then you must apply the same rubric to LLMs, which are only capable of expressions within their specific dataset (and thus not the entire corpus of data on which they haven't been trained) and even then which tend to confabulate far more frequently than human beings at even simple tasks.

edit: never mind, I guess you aren't willing to take this conversation on good faith.

Didn't this start with "Can someone tell me where your average every day human that’s walking around and has a regular job and kids and a mortgage would land on this leaderboard? That’s who we should be comparing against."

And the average person would do poorly. Not because they couldn't be trained to do it, but because they haven't.

It's obvious that the average person would do bad at the International Math Olympiad. Although I don't know why the qualifiers of "regular job and kids and a mortgage" are necessary, except as a weird classist signifier. I strongly suspect most people on HN, who consider themselves set apart from the average, with some also having a regular job, kids and a mortgage, would also not do well at the International Math Olympiad.

But that isn't the claim I'm objecting to. The claim I'm objecting to is "The average person is bad at literally almost everything," which is not an equivalent claim to "people who aren't trained at math would be bad at math at a competitive level," because it implicitly includes everything that a person is trained in and is expected to be qualified to do.

It was just bad, cynical hyperbole. And it's weird that people are defending it so aggressively.

It's obvious that 'bad at' in this context means 'incapable of doing well'.

Nitpicking language doesn't help to move the conversation. One thing most humans are good at is understanding meaning even when the speaker wasn't absolutely precise.

You and the parent poster seem to be conflating the ideas of:

- Does not have the requisite skills and experiences to do X successfully

- Inherently does not have the capacity to do X

I think the former is a reasonable standard to apply in this context. I'd definitely say I would be bad if I tried to play the guitar, but I'm not inherently incapable of doing it. It's just not very useful to say "I could be good at it if I put 1000 hours of practice in."

That's why there's the qualifier of "average person". If one learns to play the guitar well, they are no longer the average person in the context of guitar playing.

More than 50% of people cannot write a 'hello world' program in any programming language.

More than 50% of people employed as software engineers cannot read an academic paper in a field like education, and explain whether the conclusions are sound, based on the experiment description and included data.

More than 50% of people cannot interpret an X-ray.

> More than 50% of people employed as software engineers cannot read an academic paper in a field like education, and explain whether the conclusions are sound, based on the experiment description and included data.

I know this was meant as a dig, but I’m actually guessing that software engineers score higher on this task than non-engineers who hold M.Ed. degrees.

Agreed! Probably 3% of software could do it, vs 1% for M.Ed holders.

The only reason I chose software engineers is because I was trying to show that people who can write 'hello world' programs (first example) are not good at all intellectual tasks.

Which proves my point precisely that unless you’re superhuman in this definition, you’re obsolete.

Nothing new really, but there’s no where left to go for human labor and even that concept is being jeered at as a fantasy despite this attitude.

I really don't think it does, because we disagree on what the upper bound of an LLM is capable of reasoning about.

An average human may not be suitable for a given task, but a person with specialized skills will be. More than that, I believe they will continue to outperform LLMs on solving unbounded problems- i.e. those problems without an obvious, algorithmic solution.

Anything that requires brute force computation can be done by an LLM more quickly, assuming you have humans you trust to validate the output, but that's about the extent of what I'm expecting them to achieve.

Think beyond LLMs

You need to think about what comes after LLMs that look nothing like LLMs

You need to think about what robots with human capabilities, which are improving multiple times per day, is going to do.

Now add LLMs back in as your HMI

Machines have always had superhuman capabilities in narrow domains. The LLM domain is quite broad but it's still just a LLM, beholden to its training.

The average everyday human does not have the time to read all available math texts. LLMs do, but they still can't get bronze. What does that say about them?

Average humans, no. Mathematicians with enough time and a well indexed database of millions of similar problems, probably.

We don't allow chess players to access a Syzygy tablebase in a tournament.

Average human would score exactly 0 at IMO.

That’s not how modern societies/economies work.

We have specialists everywhere.

My literal last sentence addresses this

> average every day human

Average math major can't get Brozne.

Reply on news.ycombinator.com