I made ChatGPT take an SAT Test | Hacker News Reader

I made ChatGPT take an SAT Test

twitter.com

31 pointssprague15 comments

I’ve been experimenting with ChatGPT in another educational context: short essays that high school and college students often have to write for class. ChatGPT excels. I am putting my test results on the following page:

https://www.gally.net/temp/202212chatgpt/index.html

Some of the prompts and responses involve Japanese because I teach at a university in Japan. I especially want to see how well Japanese students could use ChatGPT to produce compositions for their English academic writing classes. Of the trials I’ve done so far, Test 18 on the above page is the most significant: The Japanese prompt outlines briefly a three-paragraph essay, and ChatGPT produces two very good original essays in English on the requested topic and with the requested organization.

In 2020, GPT-3 175B was already outperforming humans on SAT questions by ~20%. There are far more formal benchmarks for models now.

https://lifearchitect.ai/iq-testing-ai/

Surely outperforming some humans? I don't see how it can outperform perfect scores by 20%?

Obviously performing better than the average SAT (human) result.

Japan did something similar years ago with result in the top 20% for the national college entrance exam. The Todai robot could even take the exam all by itself and didn’t need a human to feed it questions and possible answers.

https://www.businessinsider.com/robot-beat-most-students-on-...

I would be really interested to see how it performs on GRE (The stupidest exam ever invented). I think it will do great on writing and qualitative but I doubt about quantitative part.

Edit: I actually tried quick trial to let ChatGPT take one quantitative practice exam (16 questions) using the same way the OP feed the questions. I was surprised that the result was 5/16 correct answers. This was even worse than I guessed. Indeed, this is just heuristic attempt and maybe someone (or me) will have more serious one later. If you are interested into what were the question and answers that, They are here [1]

i.e to how I fed the question was

> If x+2y= 4 and x+3y= 9. Find the correct option. Quantity A = x Quantity B = y

choices: Quantity A is greater Quantity B is greater Both Quantities are equal The relationship cannot be determined

And the answer was

> The correct answer is 2) Quantity B is greater.

To solve this problem, we can first note that the equations x+2y=4 and x+3y=9 can be solved simultaneously to find the values of x and y.

and it went to explain the steps.

[1] https://paste.elashri.xyz/s14t

Oh my.. I thought GRE was dead. I saw some professor folks talked about moving on and ditching GRE a while ago .. I guess that's very small sample size.

Most are gre optional. It is not dead but surely dying

Measuring human's capability is tricky. It often has guard effect against measurement tools. Exam is just a best-effort tool for some limited dumb systems.

What's missing from these models is everything related to visual or spatial information (that is not encoded in text). I assume that there will be eventually be something like ChatGPT/InstructGPT where part of the input data is images and or videos, with and without captions. So it would have a way of connecting the language to the spatial (and temporal).

It seems like they may need a more efficient approach though to handle the massive amount of video data. Maybe the 'MrsFormer' multi-resolution thing could help.

Another thing that could be very useful for coding without requiring visual information would be to add a whole other subsystem where this thing could actually compile/run the code iteratively and see the output.

I don't think transformers are the last invention in AI, but they certainly seem capable of getting to general purpose AI for many contexts. That and related techniques are not going to create something like a digital autonomous person though, which I think is a good thing.

> What's missing from these models is everything related to visual or spatial information (that is not encoded in text). I assume that there will be eventually be something like ChatGPT/InstructGPT where part of the input data is images and or videos, with and without captions. So it would have a way of connecting the language to the spatial (and temporal).

Is that missing though? Dall-E is the next tab over. You can image to CLIP, and image to image is not just visual, it involves that language to spatial and visual step:

"CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3."

https://github.com/openai/CLIP

Folks who've been playing with Dall-E and getting coherent images seem to be adept at prompting GPT-3 and getting coherent answers.

Hmm.. yes CLIP looks amazing. So since it has text and visual, maybe they could enhance it to perform like ChatGPT also?

what is the average SAT score if you give the test to three or four randomly selected people?

Twitter thread said that score is 50th percentile.

its interesting to consider...if the bot is supposed to complete the next word with the most likely response what would we expect? I wonder how often the most likely response is correct on sat

Reply on news.ycombinator.com