Still, I'm a fan of standardized tests. If there's a problem, we should try to improve the tests rather than doing away with them.
Here's a comment from a different branch of this thread:
> The findings in general is that much of the disparity between minorities and whites goes away for harder analogies that are more technical or scientific in nature, in fact contrary to what I expected, the recommendation of the study is to simply eliminate the bottom third of verbal questions (ranked by difficulty) involving analogies or sentence completion, as those are the questions that are most likely to involve cultural bias.
Secondly, that doesn't address differential prediction abilities. It addresses a concerning aspect of the verbal test, but is somewhat irrelevant when your goal is to predict college success. I'm not aware of differences on that metric.
That's good to hear. It's hard to know what research is still fresh, unless one is employed to stay current in a field.
But I think the bigger danger is that after a while, practicing for standardized tests starts monopolizing the curriculum at secondary schools, and teachers start being evaluated by their students' test scores to the exclusion of all else. Even 20 years ago, I remember getting herded onto a bus to some community college to take an extra "practice state proficiency test" whose score didn't count for anything at all.
Even if universities have lousy motivations, de-emphasizing that kind of thing probably makes high-schoolers' lives a little better.
One could make a test called 'big brain test' and sell it to schools to say it measures big brains, and it might be only somewhat correlated but still get widespread adoption because it fulfills a need schools have to have some sort of quantitative sorting of students, but it still wouldn't necessarily be a measure of big brains. This is sort of what the history of the SAT is.
so imo getting rid of standardized tests does get rid of a subjective rating of an applicant which is good in theory, but when you leave in all of the other subjective ratings that are much more difficult for disadvantaged students to do well on it who is it actually benefiting?
That doesn’t make a timed foot race a subjective measure, IMO.
Pointing out shortcomings of standardized tests is one thing. Advocating for eliminating what seems to be our most equal/standardized/objective/whatever measurement without a replacement seems like a bad idea to me.
https://slatestarcodex.com/2015/06/06/against-tulip-subsidie...
I mean, it's okay to use the best imperfect metric you have available; so acknowledging "the metric is not fully correlated and depends on other factors" does not mean anything, the only argument that has weight is "hey, this other metric has better correlation with The True Thing and less bias, use that instead"; if the remaining alternative metrics are even less objective and more biased than SAT/ACT tests (which IMHO is the case), then removing SAT tests won't and can't improve things, despite these tests being flawed.
to be clear, i still think the sat is better than everything else because preparation is much more legible and accessible compared to all of the other ways to stand out for college applications.
Which begs the question, who exactly do you admit to your school, and why?
In my not-so-humble opinion, however, the best way to prep for the verbal SAT is to read books. Novels are fine.
For the math, prep by taking all the math classes your school offers, and make an effort to learn all the material.
Do that, and you won't need prep classes, and you'll also be well prepared for college.
They can still take the test and get a high score.
> How are the SAT/ACT not objective tests? Are they not objective measures of how you perform on the SAT/ACT?
Most of the studies on this show that all of the cultural references in the reading passages and math word problems were things that wealthy people would have experienced in real life, but poor people would not, so it gave an advantage to wealthy kids. Also a lot of the vocab were words that wealthy kids had heard at home but not poor kids.
So while it was supposed to be a test of intelligence, it is really a test of how wealthy you were growing up.
Edit: Here is the main study from 2003 from Harvard: https://meridian.allenpress.com/her/article-abstract/73/1/1/...
Here is a writeup of someone who replicated it with more recent test scores: https://www.insidehighered.com/news/2010/06/21/new-evidence-...
MENSA accepts it for admission to their high IQ club.
Depends on how much money is meaningful to you. Time has its own costs, especially since an older teenager can work as an alternative.
And that's just the reading comprehension. The idea that the math word problems involve cultural bias that would affect performance is utterly ridiculous. "Most studies"... there is no chance that any quality research has been done to support something so laughable.
I will look at any citations you may have, but I am 100% confident that you have been duped.
That 2003 study was done at Harvard.
Here is a writeup of someone who replicated it with more recent test scores: https://www.insidehighered.com/news/2010/06/21/new-evidence-...
Well, I'll be damned. Huh.
> The 2003 study and this year's found no DIF issues in the mathematics section.
The earlier claim about math word problems being culturally biased seems unsupported by the citations.
But will the claim be withdrawn? Hmm, let's find out...
Runner is to Marathon as XXXXXXXX is to Regatta
If you grew up on the coast in a wealthy family, you have a much higher chance of knowing what a Regatta is and that it involves boat races.
It would be hard to deny that generally sailboat racing is done by wealthy people.
The yearly regatta has always been like the Olympics of Newfoundland
There are no questions like that any more. You really think the people designing the highest-profile standardized test in the country are oblivious to this kind of problem these days? There's a ton of cultural and racial bias screening and filtering that goes into producing these questions now, both manual and statistical. You can look at some sample questions yourself.
You can definitely cram for the grammar / math portions, but the reading comprehension sort of just requires a lot of reading experience. There isn't a rote set of things you can memorize to get a high score, which could arguably be seen as good or bad.
Maybe more practically, we should consider: Do lower income students read less on average? And if so, is the difference greater than their ability to gain access to tutoring services / prep books which can be used for the crammable portions of the exam?
Poverty makes things harder, no doubt. But American parenting is just bad and American students don't try very hard. My wife comes from a lower-income white family (multi-generational lack of economic mobility) and the lack of emphasis on eduction or discipline at home is glaring. Some people, like my wife, are just naturally very self-driven and make it out. But a lot of potential gets wasted because kids are allowed to just faff around instead of being rigidly guided by parents (or, as is more commonly the case, by the one parent).
The findings are that much of the disparity between minorities and whites goes away for harder analogies that are more technical or scientific in nature, in fact contrary to what I expected, the recommendation of the study is to simply eliminate the bottom third of verbal questions (ranked by difficulty) involving analogies or sentence completion, as those are the questions that are most likely to involve cultural bias.
If my understanding is correct (and I'm sure I have some subtle details wrong), the idea is to let students take the SAT, then go over the verbal/analogy/sentence completion portion of the SAT and order each question by the proportion of white students who answered it correctly, and then eliminate the easiest of those questions from the final score. The most difficult questions are still part of the test, but apparently most of the cultural bias is among the easiest questions to answer that minorities get tripped up on.
If your hypothesis were true, that it's mostly a matter of bad parenting, it's kind of hard to see why bad parenting would make minorities struggle to answer the easiest questions for white people, but not also struggle to answer the hardest questions as well.
[1] https://www.insidehighered.com/news/2010/06/21/new-evidence-...
This is all you needed to say.
The problem is that the system is designed to identify the most effective parents, not the most talented children. If you look at college admission as a function of personal merit, then it's unfair to judge a child based on their parents' ability to raise them. If you look at it as a system to identify those who will benefit society most by being educated, then you're going to get a lot of false positives that have parents from higher socio-economic classes.
Providing a test that produces an empirical result is just making it very easy for parents to game the system. It's a single, stationary target that they can fire money at. If you take away the test you're still selecting for the best parenting (all of life is). But it lets universities put more emphasis on grades and personal stories, areas where students whose parents have never heard of the SAT can still excel.
If you can use money to game the test then it's culturally loaded toward people who have enough money to game the test. A broader admissions criteria doesn't have this specific flaw
And you’re missing the point re: income, which is that all poor kids aren’t similarly situated. Asian kids raised in the bottom 20% of the income scale have a 25% chance of ending up in the top 20% as adults. For white kids it’s only 11%. Socioeconomic status is a factor but it’s not dispositive or insurmountable.
As to grades—you think it’s easier for a bright but poor kid to do their homework in time for 12 years than to study for one test? As to “personal stories”—that’s just a vector for manipulation by the whites people who run the admissions system. It’s remarkable to me to read anecdotes from admissions counselors, who say stuff like that they are looking for weird and non-conforming kids: https://www.cnbc.com/2021/10/22/former-stanford-admissions-o.... Talk about cultural loading!
I never said that someone's income quantile was dispositive. If my parents never took the SAT and have no idea what it is, I'm at a huge disadvantage to do well on the test by any measure. Regardless of their income. Again, the SAT isn't testing for academic aptitude but for the cultural knowledge of parents and peers.
As a wealthy parent I know exactly how to throw money at prepping for one test: hire expensive tutors and purchase practice tests. Gaming years of GPA or coming up with a compelling story is more complicated and difficult to solve with the application of financial resources.
A system with vague and subjective criteria is subject to bias, for sure. They are also more difficult to manipulate directly with wealth (other than bribes). We have to decide what our priorities are there as a society.
Runner is to Marathon as XXXXXXXX is to Regatta
If you grew up on the coast in a wealthy family, you have a much higher chance of knowing what a Regatta is and that it involves boat races.
But if you read the papers cited in the GGGP comment, those hard problems aren't the problems that show the racial gap. The hardest problems show the least gap. Which doesn't fit the pat little sailboat jargon theory at all. However it does fit a common pattern, also seen in longevity statistics, where the most exceptional are more similar to one another than the average are.
Wait, can you elaborate?
As people get older, the mortality rate gaps between different groups get smaller on a relative basis. Like, the ratio between rich and poor 50 year olds' risk of death over the next year might be 3x or more, but as they age, they become more similar, and by 90 the ratio may be more like 1.3x, and after 100 it becomes indistinguishable from 1 (in part because of thin data).
The theory is that the genetics and environmental conditions needed to survive to an old age are fairly specific, so once you're down to those people, they have less variation from one another. The health risk factors that distinguish income and racial groups may mostly impact younger age mortality, and the people that get past those younger ages may have avoided or resisted those risk factors, regardless of race or income.
Similarly here, students who "survive" to do well on hard SAT questions may be fundamentally very similar to one another (sharing a high g-factor), and this may greatly reduce socioeconomic and racial gaps that are seen on the easier questions.
Does this mean the easier questions are biased? Not necessarily, I would argue. The concepts tested in the easy-medium questions are most likely to be intellectually foundational -- you are far, FAR more likely to get obscure rich people jargon in the hard questions than the easy-medium ones. We want people to have the basic intellectual exposure that enables them to do well on those easier questions, and we should reward it.
The problem is not the test, the problem is racially biased failure to educate in the lead-up to the test. This explains the pattern perfectly: the strongest students power through regardless of background, while the weaker are more sensitive to resources they were given. So you see the biggest gaps on easy-medium questions where basic educational background matters most to getting them right, rather than raw talent.
Have you explored broader implications of your idea? For example people who survived tribulations of a political career may end up close to each other despite their initial difference in political opinions, and maybe even across different countries. This doesn’t sound quite as convincing, but I’m kind of grasping in the dark trying to find some other examples.
I don't know of any examples outside of health but health is a big enough topic and closely related enough to education that I think it's likely to have some application.
Certain psychometric studies also support higher sensitivity to environment in the lower economic strata. Not exactly the same but you see the connection.
The studies cited by jedberg are very interesting, I just don't think they show what he thinks they do.
A better alternative to trashing the SAT is to provide an estimated adversity score, as ETS recently began to do, so schools have a sense of what a student may have had to overcome to reach their level of achievement.
Still there is something in the longevity example that’s evading the trivial explanation.
It's also a little unclear whether we should consider the % difference convergence remarkable, given that mortality at older ages is so much higher and the absolute difference may not show much convergence.
Incredible educational resources are available online for free and just a click away on that supercomputer in everyone's pocket.
There's no reason to be restricted to your parents' vocabulary.