Hacker News Reader

I still don't understand why standardized testing gets so much pushback. Having the students do their work in a controlled environment is the obvious solution to AI and many other problems related to academic integrity.

Its also the only way that students can actually be held to the same standards. When I was a freshman in college with a 3.4 highschool GPA, I was absolutely gobsmacked by how many kids with perfect >= 4.0 GPAs couldn't pass the simple algebra test that the university administered to all undergraduates as a prerequisite for taking any advanced mathematics course.

> I still don't understand why standardized testing gets so much pushback.

Goodhart's law.

Nah. Goodhart's law is literally just "if you play a matrix game don't announce your pick in advance". It is not a real law, or not different from common sense. (By matrix game I mean what wiki calls "Normal form game[0]", e.g. rock-paper-scissors or prisoner's dilemma.)

In education, regarding exams, Goodhart's law just means that you should randomize your test questions instead of telling the students the questions before the exam. Have a wide set of questions, randomize them. The only way for students to pass is to learn the material.

A randomized standardized test is not more susceptible to Goodhart's law than a randomized personal test. The latter however has many additional problems.

[0] : https://en.wikipedia.org/wiki/Normal-form_game

That's not even remotely true. A randomized standardized test will still have some domain that it chooses its questions from and that domain will be perfectly susceptible to Goodhart's Law. It is already the case that no one is literally teaching "On the SAT you're going to get this problem about triangle similarity and the answer is C." When a fresh batch of students sits down in front of some year's SATs the test is still effectively "randomized" relative to the education they received. But that randomization is relative to a rigid standardized curriculum and the teaching was absolutely Goodhart'd relative to that curriculum.

"The only way for students to pass is to learn the material."

Part of Goodhart's law in this context is precisely that it overdetermines "the material" and there is no way around this.

I wish Goodhart's law was as easy to dodge as you think it is, but it isn't.

I do not believe schooling is purely an exercise in knowledge transfer, especially grade school.

School needs to provide opportunities to practice applying important skills like empathy, tenacity, self-regulation, creativity, patience, collaboration, critical thinking, and others that cannot be assessed using a multiple choice quiz taken in silence. When funding is tied to performance on trivia, all of the above suffers.

  They promise education, but really they give you tests and scores

  And they predictin' prison population by who scoring the lowest

For those who are out of the loop:

Run The Jewels - Walking In The Snow

> https://www.youtube.com/watch?v=6-M15L4BTqI

> https://genius.com/Run-the-jewels-walking-in-the-snow-lyrics

Well, for one thing, people learn differently and comparing a "standard" test result just measures how much crap someone has been able to cram into their brain. I compare it to people memorizing trivia for Jeopardy. Instead what needs to be tested and taught is critical thinking. Yes a general idea of history and others is important, but again its teaching people to think about those subject, not just memorizing a bunch of dates that will be forgotten the day after the test.

you cannot possibly do any higher level of analysis of any subject if you dont even know the base facts. its the equivalent of saying you dont need to know your times tables to do physics. Like, theoretically its possible to look up 4x6 every time you need to do arithmetic but why would you not just memorize it.

If you dont even know that the american civil war ended in 1865 how could you do any meaningful analysis on its downstream implications or causes and its relationship to other events.

More important than knowing what 4x6 is, is understanding what multiplication is, why division is really the same operation, understanding commutative, associative, distributive properties of operations, etc. All of this comes as a result of repeated drilling of multiplication problem sets. Once this has been assimilated, you can move on to more abstract concepts that build on that foundation, and at that point sure you can use a calculator to work out the product of two integers as a convenience.

How is knowing the commutative properties of operations more significant than being able to do basic arithmetic?

I'd imagine millions if not billions of people have found basic math useful without ever learning what "commutative" even means.

More important in the sense of being able to understand higher abstractions, not necessarily more important in a strictly practical day-to-day sense.

Every standardized test I have ever taken requires critical thinking, and none of them have been about regurgitating facts.

It doesn't sound like you've ever taken a standardized test, that's not how they are structured at all.

That's an argument against ranking students in a way which can potentially determine their entire lives, not an argument against standardized testing. The alternative to standardized testing is GPA which is an extremely subjective metric masquerading as some objective datapoint. Kids from different schools don't even have the same curricula let alone the same grading standards.

Also the teachers have a vested interest in giving the highest grades they can to as many students as they can without making it obvious that they aren't actually grading them fairly. i don't mean this as an accusation against anybody or some sort of insult against teachers as a whole, I merely mean to point out that this is what they are incentized to do by virtue of the fact that they are indirectly grading themselves by grading their students.

> I still don't understand why standardized testing gets so much pushback.

Parents of the children who can't pass the standardized tests also get a vote.

> In an age where having a degree is increasingly meaningless

I wish I would agree with you, but I think that having a degree (or rather the right degree) is more important than ever.

Basically grades exist to decide who gets a laid back high paying job, and who has to work 2 low paying labor intensive job just to live paycheck to paycheck.

As one teacher told me once: we could have all of you practice chess, make a big tournament and you get to choose your university based on your chess ranking. It wouldn't be any less stupid than the current system.

> Basically grades exist to decide who gets a laid back high paying job, and who has to work 2 low paying labor intensive job just to live paycheck to paycheck.

Rather:

Grades exist to decide who gets a stressful, but rather high-paying job, and who has to work 2 low paying labor intensive job just to live paycheck to paycheck.

Laid-back high-paying job (unluckily) barely exist.

>What somebody scored in a high school writing class should matter far less than the output somebody is capable of producing when given a prompt and an hour in a closed setting

Sure, but it takes < 1 second to read a GPA.

Right, in an ideal world we'd peer into the minds of people and compute what they know. But if we did that, our eyes would probably catch on fire like that lady in Kingdom of the Crystal Skull.

We need some way to distill the unbelievable amount of data in human brains into something that can be processed in a reasonable amount of time. We need a measurement - a degree, a GPA, something.

Imagine if in every job interview they could assume absolutely nothing. They know nothing about your education. They might start by asking you to recite your ABCs and then, finally at sunset, you might get to a coding exam. Which still won't work, because you'll just AI cheat the coding exam.

We require gatekeepers to make the system work. If we allow the gatekeepers to just rubber stamp based off of if stuff seems correct, that tells us nothing about the person itself. We want the measurement to get close to the real understanding.

That means AI papers have to be given a 0, which means we need to know if something is AI generated. And we want to catch this at the education level, not above.

I did have interviews with a government agency many years ago that, among other things, involved a battery of tests including what I assume were foreign civil service exams. I got an offer though I didn't take it.

But assuming in-person day long batteries of tests for universities and companies is probably not very practical.

You can argue whether university is a very efficient use of time or money but it presumably does involve some learning and offers potential employers some level of a filter that roughly aligns with what they're looking for.

Companies and governments use to do testing all the time. A number of lawsuits about those practices being discriminatory resulted in them going away.

No, they used to do IQ tests which were found discriminatory (and don't correlate with job performance). Tests and problems directly or indirectly related to the job are not only legal, but are commonly used (leetcode).

IQ tests are perfectly legal in US hiring, but you're right, they're not done because they don't work well.

Chicago police and fire had a lot of problems with their entrance exams and promotion exams. Don’t know if they were strictly IQ tests.

In a world where some but not all programs are “diploma treadmills,” you would expect that the reputation of the bad credentials would go down and the good credentials would go up. In some sense if the credentials were really being used (and not just as a perfunctory first pass elimination), you’d expect the most elite programs to have the highest signal to noise ratio. But the market doesn’t seem to respond to changes in credentialing capability (by hiring more from programs that start focusing on the “right” things to test). Instead it’s really just a background check.

In the US, it's also because there are so many options that it's not feasible to have a clear ranking of schools outside of the extreme ends of the spectrum.

> you would expect that the reputation of the bad credentials would go down and the good credentials would go up.

We should expect this if employers can efficiently and objectively evaluate a candidate's skills without relying on credentials. When they're unable to, we should worry about this information asymmetry leading to a "market for lemons" [0]. I found an article [1] about how this could play out:

> This scenario leads to a clear case of information asymmetry since only the graduate knows whether their degree reflects real proficiency, while employers have no reliable way to verify this. This mirrors the classic “Market for Lemons” concept introduced by economist George Akerlof in 1970, where the presence of low-quality goods (or in this case, under-skilled graduates) drives down the perceived value of all goods, due to a lack of trustworthy signals.

[0] https://quickonomics.com/terms/market-for-lemons/

[1] https://competitiveness.in/how-ai-could-exacerbate-the-skill...

> But the market doesn’t seem to respond to changes in credentialing capability

I mean, it certainly seems to. I've been in hiring roles in tech for 20-ish years, and have definitely seen changes in how college hire patterns based on credential values. Some schools have gone way up in how much we value their credentials (Waterloo), some have gone somewhat down (MIT), etc.

> This is one of the very few places I think big tech in the US has done a great job. Coding interviews can be justifiably critiqued in many ways, but it's still a much better system than raw credentialization.

Just so we're clear, the coding tests are in addition to credentialisation. I'll never forget when I worked at Big Tech (from Ireland) and I would constantly hear recruiters talk about the OK school list (basically the Ivy league). Additionally, I remember having to check the University a candidate had attended before she had an interview with one of our directors.

He was fine with her, because she had gone to Oxford. Honestly, I'm surprised that I was able to get hired there given all this nonsense.

If anything, it will get worse. There was a deficit of tech workers, from now on, there will be an excess. Which means that differentiators will be even more important.

The smaller the job market relative to candidates, the more the differentiators are both important and also meaningless to the actual job

We gotta play the game, though.

My experience with big tech has been the polar opposite - nobody has ever cared and I've never tried to hide it either. Which one was it if you don't mind me asking?

I'm a drop out (didn't finish BSc) from a no name Northern European university and I've worked at or gotten offers from:

- Meta

- Amazon

- Google

- Microsoft

- Uber

- xAI

+ some unicorns that compete with FAANG+ locally.

I didn't include some others that have reached out for interviews which I declined at the time. The lack of a degree has literally never come up for me.

It was Meta but twas a long time ago, when they were known as Facebook.

It seems to be a US role thing in my experience.

Hiring is still a pretty non-uniform thing despite attempts to make it less so - I'm sure there are some teams and orgs at all these large companies that do it well, and some that do it les well. I think it is pretty well accepted that university brand is not a good signal, but it is an easy signal and if the folks in the hiring process are a bit lazy and pressed for time, a bit overwhelmed by the number of inbound candidates, or don't really know how to evaluate for the role competencies, I think it's a tool that is still reached for today.

In a way, I think the hiring process at second-tier (not FAANG) companies is actually better because you have to "moneyball" a little bit - you know that you're going to lose the most-credentialed people to other companies that can beat you dollar for dollar, so you actually have to think a little more deeply about what a role really needs to find the right person.

Once you have a relevant work history, a degree matters much less. It still does to some employers, however, for whom it's a simple filter on applicants: No degree? Resume into the bin.

It is true, and it is much easier to get that first relevant work experience when you have a big degree. Not impossible without one, just much harder.

That's an absolutely dreadful exam. Would you mind explaining the point you wanted to make with it?

It's a product of its time, but what exactly do you think makes it bad?