Hacker News Reader

Comment by sublinear | Hacker News Reader

I think this is evidence against the usefulness of standardized tests rather than support for ChatGPT

There is a leap in this logic- going from the fact that SAT is solvable by a LLM to the conclusion that the SAT is useless. Questions about addition can be solved using calculators, does that mean that teaching addition is useless? Also there is an underlying assumption that memorization is a useless skill, which I'd argue against, as in today's world most innovation is done by identifying patterns between disciplines, and memorization is an important part of recognising patterns

not so much memorization as decomposition of numerous examples into logic based knowledge. As a simple example, knowing volumes of words is not as useful as knowing etymology since etymology brings you a deeper structure.

I'd say that's subjective, depending on context and individuals. Sure etymology might be amazing to know for a lit major or author, but for someone who wants to just be able to write clearly, knowing a lot of words is useful.

Well it shows that you can do well on SAT with basic word association without understanding and explaining the ideas of the passage.

The "Joe Bloggs" theory has known this for ages.

Yeah but that also says nothing about how useful or useless that is

You're correct that memorization and pattern recognition are useful, but they're not supposed to be the majority of the test!

A good test should also include reasoning skills and comprehension.

Having reasoning skills means not all necessary information can be derived from the test itself, the necessary information isn't always well defined, yet a correct answer is produced.

Demonstrating comprehension requires the test taker to write out a model of what they think they know in their own words. Undoubtedly the answer will be at least slightly wrong which is why subjective expertise is required to evaluate the level of comprehension.

To me it seems the quality of the tests has deteriorated due to cost cutting, not because AI has become so much more sophisticated.

Subjectivity is usually dismissed these days as a roll of the dice, but it's vital to fitting the real world full of humans living in it. Humans must be the gatekeepers, not to fight off AI, but by the very definition of what it means to be convincing.

Answers to real world questions solve many more constraints than just the concrete subject matter. The whole point to updating a test on a well understood topic should not merely be to prevent "replay attack" cheating, but to conform to the new subjectivity on the topic.

What I find more concerning is the 100% scores that ChatGPT has been able to achieve on AP CSA and APUSH, among others.

It makes sense to teach 3rd graders 2+2, but if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools.

So many AP courses are just memorization and identifying formulas. I'd bet the majority of AP CSA students wouldn't fare very well when tasked with detailing how they'd approach a project, whereas kids in IB schools etc. would find it straightforward.

I don't find it at all concerning that e.g. a history course is primarily tested and easily passed by memorization. There's nothing wrong with teaching context instead of pure skills all the time IMO. I do agree it'd be good to see deeper computer science as an early option in the US but at the same time I'm not sure whether or not something like ChatGPT can pass is really a good indicator for why.

The AP CSA pass surprises me (I've never taken it myself). I had this thought because I tried to use chatgpt for work programming stuff and it got stuff wrong very frequently. So frequently, and to a great enough degree, that I had to keep re-circumscribing what kind of problems were worth my time to throw at it, and now it's a tiny cross-section.

Thats what surprises me as well. If I give chatGPT anything that requires actual thought or reasoning put into it, it just produces gibberish.

It's great for getting a blueprint to start from. I recently had to write a gtk+rust client, and I asked chatGPT to give me some starter code which largely worked.

All the chatGPT hype feels the same to me as the copilot hype from a while back. The people who got the most excited about it are the same people who its going to replace.

That does seem like a good application for it. If one thinks something is in the corpus many times (various GTK idioms), chatgpt is pretty good at a type of syntactic composition...e.g., "can you show it to me in rust"

There is one argument against this though that is somehow brewing in my brain.

The end goal is to internalize the subject so that you can extrapolate your own insights.

"This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples.

And to reach that kind of insight you need to start filling the brain with information. That may be called memorization. The memorization is just one path of the road towards the black belt. If you skip it you will be lost.

Yes, but ... what this shows is that testing for "memorization" isn't good enough. We all know people who have learned information and can give answers with confidence, even though they are just winging it, to put it politely. ChatGPT could be considered their equal on such tasks. So schools need to test that students are capable of working with the acquired information in a way that befits the level of the course, just like they need to find a way to lose their addication to the lazy "write an essay" assignment. If not, education will lose its status.

The question is if we should expect these test to test on the black belt level or at a lower level belt?

(i am not native English speaking so i don't know what addication means, I couldn't quite understand that sentence)

Seems to be a typo for Addiction

Sorry, that was a typo for addiction indeed. They should test at the level they claim to certify. So, an MSc. should not get questions that can be answered by rote, but it might be appropriate for a first-year student, when the topic is entirely new. And teachers should read essays for depth and understanding, instead of counting the words and skimming a few paragraphs. That'll draw complaints from the people that think every end goal and test of a course should be perfectly described and objective, but to them I say: move your pretty, empty heads elsewhere if you can't see you're parroting someone's failed attempt to operationalize a remote ideal.

There is a whole question about how useful it is to test students. In my experience, some will understand and learn the material (which is the goal) and successfully pass the test, while others will just learn how to pass the tests.

Now it would be wonderful if we could just motivate students to understand and learn the material, without putting so much pressure by testing them all the time.

> "This happened in WWII so if these conditions starts to appear it risks to happen again".

There is a reason why history repeats. This kind of learning never happens in human society.

We currently have similar warning signs to WWII, but nobody cares. We are currently at the stage where if you express an opinion that does not agree with the approved narrative, you get downvoted/cancelled. But nobody cares.

I care, and suffer, all the time. It feels like I am doing the right thing but maybe the masses know better. There's safety in keeping your head down

Part of the history exam is just that, internalizing the subject.

The part that ap fails to deliver on is the written portion, where it asks you to extrapolate your own insights. If chatgpt can solve the problem that should require human insight, is the prompt really testing students properly?

> This happened in WWII so if these conditions starts to appear it risks to happen again". Just to take history as an example. All other subjects have similar examples

History is a particular bad case to drive such example since there are thousands of factors that are always different and we have very poor data beyond a few centuries so theres that. With History you are much more likely to end up with confirmation bias than actual understanding.

> if college board is creating "college level courses" with problems that can be easily solved through whats still a fairly new technology, we need to reevaluate what we're teaching in schools.

I don't get it. AP calculus is problems that can be easily solved with what is now very old technology. The point of an AP test isn't that the problems are so difficult.

These specific subjects shouldn't be solely about memorization, rather they should teach kids how to link that information together to solve problems (or in the case of history, avoid them in the future).

Thats the issue. The point of an AP test shouldn't be that the problems are difficult. But currently thats what it's all about. They're just trying to test that kids are capable of surviving a "rigorous academic curriculum".

[1] https://apcentral.collegeboard.org/about-ap/ap-a-glance

Why? This section is literally just questions about a modestly complex reading passage. It’s the most real world part of the SAT.

Also you’re not allowed to use any reference materials and certainly not the Internet when you’re taking these tests.

These particular tests maybe. I'd want to see how it does on the ones from a few decades ago.

Reply on news.ycombinator.com