You don't get exactly the same test in the end, similar with SAT, but the constraints we put on these tests (they have to be comparable) produce patterns in the questions you can train for. This is the same logic why people can train to improve their SAT scores, if they were a measure of true innate intelligence their training would have no impact on their score.
I would presume not - most tests are timed, and if you are spending time on first-time only tasks in understanding the problem then the result is inaccurate. If you train out those first-time tasks so that you are repeatably using the time budget in the test to solve problems then you should reach some kind of steady state and produce repeatable and more accurate test scores.
My take is that the repeatable scores measuring your steady state in the task would be more accurate than the untrained scores with an unknown amount of initialization time within each problem. I would make a similar claim to naasking below that this could account for some of the Flynn effect.
Not every person takes a SAT prep class to improve their test score. There are lots of people who are truly above average in terms of intelligence and can score very high on the first try.
First, I don't think that's strictly true. Obviously they wouldn't do as well but they still do better than chance.
Second, there's evidence that this is a big part of the Flynn effect, which means humans are susceptible to a similar phenomenon.
Your original reply insinuated that the AI is learning very similar to how humans do and that’s just not true. Yes, humans do pattern matching based on prior experiences/knowledge like AI does when you train a model, but human intelligence goes way beyond that.
Humans are trained on orders of magnitude more multimodal data over their lifetimes. Also, humans are not borne as an unbiased model, billions of years of evolution have crafted many implicit biases into our cognition (like a propensity to language, facial recognition, etc.). All machine learning models are true blank slates, so it takes a lot more data just to build up to the same starting point as a newborn human.
All that's to say that you have no basis upon which to claim that AI learning is NOT similar to how humans do it, or that human intelligence "goes way beyond hat", it's just that humans have a head start and a lot more data to work with.
What in the world are you talking about? I must be talking to chatgpt and am done with this thread. We were originally discussing the differences in methodology between AI and humans for passing standardized exams. Those involve tasks like applying well-defined mathematical concepts to a brand new problem, not “multimodal” data or facial recognition.
You don't have to point out failure modes of GPT, I know what they are. The question we're discussing here is what this indicates, if anything, about how these systems operate as compared to human brains, and whether the differences come down to training data or the fundamental architecture.
So isn't this literally moving the goalpost? "So what an AI can beat the SAT, so can humans"
Prompt:
I'm talking with a user named "331c8c71" on Hacker News. According to statistics I found by Googling, the number of Hacker News readers living in the USA is 36%. Assuming this is correct, what is the probability that 331c8c71 does not reside in the USA? Please explain your reasoning.
Response:
If we assume that the statistic you found is accurate, then we know that 36% of Hacker News readers live in the USA. Therefore, the probability that any given Hacker News reader does not live in the USA is 100% - 36% = 64%.
However, we do not know if 331c8c71 is a typical Hacker News reader. They could be part of a demographic that is more or less likely to live in the USA. Without more information, we cannot say for certain what the probability is that 331c8c71 does not reside in the USA.
So, the answer is 64% assuming that 331c8c71 is a typical Hacker News reader, but it may not be an accurate estimate depending on additional information about the user.
---
Like I said, these models are far from 100% accurate, and there is lots they get wrong, but they clearly are capable of some kind of reasoning that goes beyond simple text substitution of training data.
It is an incredible achievement that LLMs produce human-like output (e.g., wouldn't know if a gpt bot answers me unless we are discussing a topic where precision/accuracy are important) but they hallucinate (they are confident BS-generators).
The hype is that LLMs can solve any problem and replace humans (jobs). It is not so.
It may depend on what you do but I find it is easier/faster to do the work myself then to spot and fix [a possibly subtle] error in AI output. Though some of the specific things will improve in time and you can find tasks where AI is useful even today.
I don't see how the models can improve for general tasks (AGI) without being existential threat to humans (not just jobs).
In particular, LLMs fail miserably at tasks like "apply this simple pattern many times in succession" aka "for-loop", because they can't count in an abstract way, only on concrete contexts.