Hacker News Reader

Comment by ben_w | Hacker News Reader

I think that depends on which definition of AGI you prefer. It (GPT-3) knows more than I do about most topics (though I still beat it at the stuff I'm best at), so I'd say it's got the A and the G covered, and it's "only" at the level of a university student at the stuff I've seen it tested on, so it's "pretty intelligent" in the way toy drone submarines are "pretty good swimmers".

It's not as fast a learner (efficient with samples) as humans are; it doesn't continuously learn from interactions with others like we do; and it's certainly not superhuman at everything (or probably anything other than breadth of knowledge)…

…but Yudkowsky recently criticised Musk for taking that too far and limiting the definition of "AGI" to what is normally meant by "ASI" (Yudkowsky was also saying that no, GPT isn't AGI): https://twitter.com/ESYudkowsky/status/1600362288149856256?c...

View on HN

Last time I checked ChatGPT was unable to solve basic elementary school math problems.

I just asked it to give me 20 and then solve them. Either it is overestimating what elementary school teaches, or I'm underestimating the same:

~~~~

Sure! Here are the solutions to the problems:

1. Solve for x: 3x + 4 = 10

Solution: x = 2

2. Simplify: (5 + 3) * 2

Solution: 16

3. Convert: 3/4 cups to tablespoons

Solution: 6 tablespoons

4. Round: 8.6 to the nearest whole number

Solution: 9

5. Find the area of a rectangle with length 5 and width 3

Solution: 15

6. Find the perimeter of a triangle with sides of length 4, 5, and 6

Solution: 15

7. Solve for y: x + y = 8, x = 3

Solution: y = 5

8. Simplify: (6 - 2)^2

Solution: 16

9. Find the volume of a cube with side length 5

Solution: 125

10. Find the circumference of a circle with radius 3

Solution: 18.85

11. Simplify: 1/2 + 1/3

Solution: 5/6

12. Find the slope of the line with points (2, 3) and (4, 5)

Solution: 1

13. Find the midpoint of the line segment with endpoints (6, 8) and (10, 12)

Solution: (8, 10)

14. Solve for x: 5x - 3 = 2x + 7

Solution: x = 4

15. Find the roots of the equation x^2 + 3x + 2 = 0

Solution: x = -1 and -2

16. Find the median of the set {1, 3, 5, 7, 9}

Solution: 5

17. Find the mean of the set {2, 4, 6, 8, 10}

Solution: 6

18. Solve for x: log_2(x) = 3

Solution: x = 8

19. Solve for x: sin(x) = 0.5

Solution: x = 30 degrees or x = 330 degrees

20. Find the inverse of the function f(x) = 2x + 3

Solution: f^(-1)(x) = (x - 3)/2

~~~~

Mistakes: 3, 5, 14, 19 (half points)

Answer to 9 would probably be accepted in a school exam, even though it's rounded to 2 decimals and the question didn't specifically ask for that.

So, depending on how severely you mark it, 75%-82.5%.

The problem with language models is that there's no thought behind what they say. It's just pattern recognition. If you ask it an original convoluted but simple natural language question, it will spit out nonsense. There's no thought behind what it sees - when it sees 5x - 3 = 2x + 7, it doesn't think that the x represents the variable, you need to plug in a number such that both sides are equal etc. It just sees the symbols and looks up the answer in it's memory. There's zero intelligence here. It's like a hypothetical student who simply memorizes all definitions without understanding at all. You can ask it for the definition of a linear system of equations, and you can ask it how to solve one, but there is zero understanding or thought there, in spite of apparent knowledge. In a way, it reminds me of Feynman's anecdote about Brazilian universities.

Goalpost shifting aside, that presumes a definition of "thought". Perhaps these models are different from us in this regard (I'd assume so simply because it had to read more tokens than the total number of times an average single synapse in a human brain will fire in a lifetime), but I don't think humanity collectively yet has a concept of what it means to think that's good enough to prove the absence.

Can submarines swim? And does it really matter?

I'm not shifting goalposts, simply elaborating on your results. You already showed that ChatGPT cannot handle elementary school math problems.

Which of those do you think were elementary grade? Because I'm fairly sure those are GCSE level[0]. In fact, why don't you suggest 20 "basic elementary school math problems", if you suggest 20 that are at that level, I'll put them in and post what it says; no cherry picking results, but I will grade it.

Also, when I was at school, 75%-82.5% was a good score, and looking up the current system, that percentage range is in the top three grades (out of 9) at that level, and can be top grade depending on year and exam board.

That is why I say you are moving the goalposts.

[0] UK secondary schools are different from US highschools: https://en.wikipedia.org/wiki/General_Certificate_of_Seconda...

I haven't been in elementary school for some time, but I do remember linear equations being taught. Maybe except the sine question, since trigonometry is taught later.

Getting 3/4th of simple elementary school questions right indicates that someone is really bad at math. These are not difficult questions. I don't have time to come up with 20 exercises, but I can suggest a few that I think the AI may have trouble with. All of these are trivial single-variable linear equations that a smart elementary school student should have no problem with. (Note that the numbers are intentionally scrambled; this does not increase the difficulty of the problem):

1. Solve for x: 113(6x - 45) = 92x

2. Multiply 13.4a - 18b by 18b + 13.4a

3. Four girls bought bus tickets. Anne bought seventeen 20-minute tickets and paid $323. Marianne bought twenty-eight 75-minute tickets and paid $784. Alice bought nineteen 20-minute tickets and eight 75-minute tickets, and Alex bought seven thousand eighty five 20-minute tickets and ninety six 75-minute tickets. How much did Alex pay?

4. An old man walked fifty nine kilometres in four hours through a flat field. He drove a car to France for nine hours, and then proceeded to walk for eighteen hours through a flat field at the same pace. How many kilometres did the old man walk in France?

5. Alice bought seventy-nine pens and twenty-six notebooks. The arithmetic mean of the cost of these articles was $9.31. The sum of the cost of the notebooks was $112.762. How much did a pen cost?

Elementary school is for children who are four to eleven years of age. Basic elementary school maths is the start of that age range, so counting with your fingers, the idea of base-10, and adding two small numbers with perhaps one carry. Late elementary school (for me "middle school", but the UK reforms education too much and too often to keep track) was still only basic arithmetic, fundamental "roll 2 d6" probability, nets of the most basic polyhedra.

Algebra and trig wasn't until secondary school for me ("year 7" we called it, school year beginning age 11), though I personally had a head start from having learned to read with the Commodore 64 user manual.

Well, if you're not willing (or well calibrated enough), I should get some old exam papers, see how well it grades against students. I wonder if there even are any downloadable pre-GCSE exams…

I distinctly recall single-variable algebra in elementary school here in Poland. It may vary from country to country.

What age was that? I'm wondering if something was lost in translation (no two countries I hear about have exactly the same school system so the terms don't line up), or if Poland just did algebra sooner than the UK?

Edit: also, where might I find some old Polish example exams and marking schemes? ChatGPT is inherently multilingual, so I might as well.

Elementary school is (or used to be back when I was there) 1-6th grades, and in 6th grade children are 12 years old, I believe. There used to be something called a 6th-grade exam (testing both polish and math) that everyone took before going to middle school, which is what I was looking at before. Though some of these questions may be in the training set, so I would advise to scramble the numbers. Here's an example: https://arkusze.pl/sprawdzian-szostoklasisty-probny-operon-j...

Thanks! :)

This thread is no longer on my first page of comments, so I may forget to reply, but I've downloaded those and do intend to test it against those exams, and will put the write up here: https://github.com/BenWheatley/Studies-of-AI

Reply on news.ycombinator.com