How Asian test-prep companies swiftly exposed the brand-new SAT | Hacker News Reader

How Asian test-prep companies swiftly exposed the brand-new SAT

reuters.com

42 pointsphsource82 comments

Why don't they try and develop a model that takes into account that students will study directly against the test?

The current model seems to rely on what could be called "security by obscurity". Of course students shouldn't be able to spend hours analysing a single text before the test, but if you publish a corpus of 200 texts, then anyone who read and analysed them thoroughly before the test aren't really cheating anymore, they are just educating themselves.

No multiple choice test can protect against a list of questions and answers.

America uses a multiple choice test for university admission?

Both the SAT and the ACT are mostly multiple choice, with one essay.

Alongside high school transcript, essays, extracurricular activities, and demographics, yes. It's usually not a simple threshold or ranking, but it gives admissions officers a normalized view of something approximating intellectual horsepower (rigor and quality of high schools varies enormously and mostly responds to socioeconomic status of the neighborhood).

SAT and ACT each have a timed handwritten essay component, but it's not very well-regarded. There is no long-form working of problems, though that does appear on AP/IB subject tests which are also necessary to be competitive at highly selective schools.

The american education system seems obsessed with standardized tests.

The Amateur Radio license test in the US is a random selection from a large published set of multiple choice questions. Studying for the test can consist of learning the answers to all of the questions. It's very educational.

http://www.arrl.org/question-pools

I don't know if something like that would work for the SAT, but if the question pool were large enough, it would.

That's meant to be a knowledge test. The SAT isn't.

Than what exactly is it testing?

"Exactly": test taking skills.

Less exactly but still useful: critical thinking, intelligence, ability to solve problems quickly.

http://www.slate.com/articles/health_and_science/science/201...

Scholastic aptitude (SAT === Scholastic Aptitude Test) - learn random and possibly useless things and pass a test on them.

Maybe we should come up with a new testing model then. As it stand the current model is akin to a game show like Who Wants to be a Millionaire.

It correlates pretty well to intelligence. What specifically do you object to? Just the multiple choice nature of answers?

http://www.slate.com/articles/health_and_science/science/201...

"Never memorize something that you can look up."

Intelligence is dynamic and amorphous and is not something that can be measured on a single linear scale. My biggest objection to the current paper-and-and-pencil-and-no-technology test taking model is the inherent Luddism which stresses an obsolete method of learning.

Have you taken the SAT? Your description is inaccurate. You can use calculators on the test, and most questions are analytic, not the kind you can look up.

What evidence do you have for your claim about intelligence?

The calculators you are forced to use were invented decades ago. The students possess a calculator that is billions of times more powerful in their pockets but are not allowed to use them. That is Luddism.

Edit: correction: the TI-82 has a 6MHz processor where as an iPhone has a 1.4 GHz processor

What exactly do you want them to use a stronger calculator for?

Also, you're wrong about the billions of times faster. Look up the numbers.

Why should students be forced to use such outdated technology?

If we are trying to teach students how to work in the real world, shouldn't we let them use computers that they would be using in the real world as opposed to obsolete computers that were made before they were born?

The reason they only allow calculators is to avoid cheating. The kinds of problems given don't depend on complicated calculations.

>If we are trying to teach students how to work in the real world

The SAT is not about assessing knowledge. It's supposed to be an aptitude test.

>The reason they only allow calculators is to avoid cheating.

Ah this appears to be the root of the problem I was looking for, the game rules.

The question begs: what would we rather have our children have aptitude in, closed and isolated paper-and-pencil exams using obsolete technology, or open and collaborative problem solving using modern technology?

You're still missing the point that the SAT is not meant to be a test of knowledge or skill.

Notice my use of the word aptitude above

SAT/ACT are specifically meant to test intellectual aptitude; recall (aside from some probability formulas) doesn't really appear.

Saw an interesting proposal on HN a few months ago WRT entrance exams:

My university just allows anyone into first semester, but then has quite hard courses and exams there already – by the end of the second semester, 90% have already failed.

That’s another way to solve the >10x applicants issue.

Seems like a great way to reduce administrative overhead, judge based on merit, and get students to realistically self-select a college (if you don't, you waste a semester and the associated costs). Problem is that the exams select only for people that are good at taking exams - not always the most interesting/well rounded students.

Germany has such a system in place, but not without limits. How important your Grade is depends on the subject you are studying, how popular it is and how well grades correlate with success. For example, you can not be sure of a spot in a medicine school unless you have neat-perfect grades, whereas engineering or practical sciences are nearly a free-for-all, with 90% leaving after the first semester.

I don't believe a purely merit-based system could work on a large scale with popular subjects such as medicine.

Sounds like how medical school in France works, or at least did while I was there. Anyone with an "academic" HS diploma (40% of people or so) could sign up, but the overwhelming majority failed the first year. Their options were repeat the first year or go into a less prestigious track like nursing

This doesn't sound like a great idea for the U.S. system. Students would have to pay tens of thousands of dollars and spend many precious months for an experience with a 90% failure rate. I have a hard time thinking that anyone with other options would accept that proposal. On top of that, even if you're in the select 10%, you're attending classes where almost all the other students are failing. I can't imagine that class discussions are going to be helpful for you.

If this school was free or very cheap, maybe it's an option for some, but with the way things work in the U.S., this seems more like a scam than a college.

I agree. The University of South Carolina (and maybe more) has a nursing program.

They let you into "lower division" for 2 years before giving you a test to gain admission to "upper division."

Fail that test and you just wasted 2 years. Worst is that there are limited spots, so qualified people still get cut.

In many ways this sounds identical to the process of becoming a medical doctor. Fail (to either perform or get lucky enough to get a slot) at any stage and you wasted X years + Y amount of money.

This is already the case at an ordinary large state school. Not 90%, but around 50% will wash out before sophomore year. It is the very good private schools where the percentage of freshmen who will go onto graduate in 4 years is in the 80s and 90s.

I argued in a similar thread that universities should use tests just as a minimum filter. Sample equally from the top n% of scores. If you just take the very top, you just get people who have trained for that specific test, not necessarily smarter students. Or even self motivated students if their parents put them up to it.

But it really does suck that testing is so unreliable. Like for those merit based certification ideas, where they want people to pass a standardized test to get certified for a job. Instead of going to an expensive prestigious university to prove the same thing.

This is how the Harvard Extension School works. Unfortunately, it's backfired on them a little, especially given the staggering admissions process of the other schools in the university, and it's commonly known as "the school that anyone can get into" But passing the first three classes and actually being admitted into the program is very difficult. Most people just use it to take a class or two and leave.

> We’re working against cartel-like companies in China and other countries that will stop at nothing to enrich themselves,

That's how I felt about College Board.

I still don't understand how then can call themselves a non-profit while charging ridiculous prizes and monopolizing the standardized test market.

Given that Ikea is legally run and registered as a non-profit charity, I think that the term is fairly meaningless at this point. We can only judge them by their actions, just as we do any other company.

Not for profit and non-profit are actually different in different tax and law scenarios. I'm not going to try and butcher the explanation, but it pretty much boils down to how employees get paid and expenses get deducted- which, in my opinion, has made me look at not for profit companies as companies that want to appear nicer than they really are.

That sounds like a US tax law distinction, but the IKEA structure is non-US (as are the owners/founders).

they are super sketchy. https://en.wikipedia.org/wiki/College_Board#Criticism

Also, helping a handful of calculator companies maintain a monopoly on the graphing calculator market.

I ironically thought the same thing when I passed over that sentence. ETS especially.

It is time we abandon SAT and ACT tests.

SAT II's are more predictive of college performance on their own than 4 years of high school grades, but standardized tests like these take a fraction of the time, as opposed to years of intimate teacher evaluations.

The compromised operational security of these exams don't stop them from being more predictive than teachers. So if we want to retire these tests, then we need something better. What we should really be talking about is improving these tests and beefing up operational security.

Serious question; on what basis is "how well they will do at college" the right metric to use for selecting who to accept?

If you run a college then selecting people who will do well should maximise your graduation rate. Of course if this is what should be optimised is an open question.

Or you could keep graduation rate constant and make the taught material harder.

Which will lead to a revolution if you have paid colleges.

People apply, pay for a year or two, inevitably fail, but are now also 30k in debt.

What do you think those people will do?

Accepting anyone only works in countries without paid universities.

>on what basis is "how well they will do at college" the right metric use for selecting who to accept?

Well, the underlying basis is the finite number of seats for students. If you have a limited number of slots, you must reject some applicants. Therefore, the question flipped around is, "what criteria to use to reject applicants?"

That leads into what I guess your usage of "basis" is actually about: the philosophy of selecting on the "doing well" instead of some other criteria such as... "economic disadvantages?"... or "local student over out-of-state or foreign student?"..."athletic abilities?", etc. You'd have to expand on what alternatives you had in mind.

I think the "doing well" criteria can be defended by filling up the limited seats with students who will get the most out of college. For example, a student taking "Physics I" needs to know what derivatives are (Calculus). Having some slots used up by students that should have been in remedial math courses (Algebra) instead of flunking out of PHYICS1 is unfair to the potential students who already have the baseline math skills required for solving physics equations. Multiply that example across the entire curriculum required for a degree and the "doing well" criteria seems justified. You want to reject potential students who are not able to succeed at college-level course work. Maybe tests like SAT are bad/flawed but so far, they appear to be the "least bad" tool to predict which students will do well in college.

One can create institutions that don't care about the assessment of "doing well" as an entrance criteria as long as the students pay the fees. That signal is provided by many colleges[1] that don't require SAT/ACT scores. However, the prestige and respect of those institutions don't match the ones that do. That negative perception in the marketplace is not uncorrelated to their acceptance criteria. (E.g. University of Phoenix.)

[1]http://www.collegexpress.com/lists/list/colleges-that-do-not...

An excellent question.

In practice, that's not who we select. One college in the Midwest I know of selected 1) kids related to alumni, 2) people of a certain religion, 3) people who would add to the sports and music programs, 4) kids of color with good grades and high SAT scores, and last 5) people who didn't fit the above, especially from California, Texas, or Florida, who could pay full tuition. A major goal at some small colleges is balancing kids who will pay full price and kids who will do well at college, which creates a funny situation where you've got a well off heavy-majority-white party crowd with low test scores and a smaller and more diverse group of subsidized kids who bring many academic and non-academic talents (the music/sports/math/science/high school leadership crowd that may not have well-off parents).

Flip it around; how is it good to accept someone to college who does not have a good chance of doing well? How good is it to accept their money, probably put them in debt (even if not the catastrophic $100,000+ debt, even $20,000 debt can be quite a lot for a young person, and it's hard to get out of 4 years without at least that much), and claim very significant portions of some of the most important years of their life [1] if you already have good reason to believe they are unlikely to do well? (And remember we're judging lots of people so statistical bases are all we can use.)

It may not be the only criterion of use, but "do we have good reason to believe they will succeed in college?" is at the very least a critical bar that ought to be passed. It's can be the difference between college as the empowering, education experience it's supposed to be, and college as the meat-grinder wealth-extraction tool it can be when you feed in students who shouldn't be there.

[1]: Not necessarily "best", but being "earliest" in a world of compounding interest makes them quite valuable in their own way.

It's also true that the better less-selective (or non-selective) colleges provide more help than very selective colleges. I went to community college for a year and then went to a selective private college. The community college had far more help sessions, tutors, and study aids available. There was even a program at the community college that specifically sought out first-generation students to make sure they were getting extra support.

I can try: universities differ in rigor and students differ in capability. A student gets the most out of education (and becomes more capable) when presented with a challenge just below his threshold. Put him in a program that's too hard, and he'll have a shitty time while learning nothing and wasting a lot of money before washing out. Put him in a program that's not hard enough, and he'll succeed within it but his growth will be suboptimal.

Match students to the colleges that will be hard enough to get them to grow, but not so hard that they fail.

It might not matter what is on the exams. SAT could just be testing for test-taking ability. I'd assume test-taking ability would be a good predictor of college performance too, as exams are typically 40-90% of college grades in my experience.

Is there any data out there which tries to measure test skills independently?

What do you propose are the factors behind test-taking ability between-subjects? I would propose that a major contributor is g, or general mental ability, and I propose it is for that reason that the SAT II's have a non-trivial relationship to college performance, more than 4 years of high school grades.

They don't test for test-taking ability, they test for wherewithal to enroll in test prep classes.

I can't speak to the current state of affairs, but in the mid 90s, taking a 5 week Kaplan course in my peer group raised scores from the 1150-1300 range to the 1300-1500 range. That meant the difference between average state school and high probability of hitting a "reach" school.

Many of the ultra achievers on these tests start prep at age 10 or younger.

That just means that we should think about accessibility issues, but for whatever reason, teachers as a whole, despite intimate access for 4 years to students, are still less predictive of future performance than a test developed from afar.

On my phone so I don't have the paper handy, but all available research shows no more than 50 point gains on SAT from test prep.

But is that Chinese prep? Cram school is quite different from what we call prep in the west.

Well, no, since we're talking about the SAT it is pretty US-centric. Unfortunately, Duke has broken broken the link to the paper (the entire subdomain it's on appears to have gotten the redirection treatment), but [here](http://marginalrevolution.com/marginalrevolution/2014/03/the...) is an article about it.

I'm imagining a test-taking skills test. You have multiple choice questions, drawn from an incredible range of domains that people couldn't possibly be familiar with most of. I'd love to see how predictive this would be!

There's a multiple choice quiz in which the questions and answers are all basically gibberish yet you can work out the answers based on "test wiseness":

https://edtechdev.wordpress.com/2007/10/11/test-wiseness-wri...

"standardized tests like these take a fraction of the time"

Preparing for standardized tests ruins a fraction of life.

Is this a criticism, or a positive via the implication that high school ruins a much large fraction?

SAT's are not used universally, nor are they the only factor in determining acceptance when they are used. Nor SHOULD they be.

Whatever the case, it is not surprising that sketchy cram schools are cheating.

Whatever is the most performant set of predictive instruments should be used as a factor for acceptance for colleges seeking to optimize on whatever they feel are the "best" students.

College essays? Sure, if the instrument is performant. SAT II's? Sure, if the instrument is performant. Grades? Sure, if the instrument is performant. Or extra-curriculars, or whatever.

"Whatever" is certainly the case.

College admissions at many of the better schools don't simply try to get "all the best" students that they can. They look to balance many factors, including admitting some fraction of students who have connections or can pay full tuition, including taking a risk on some students who would otherwise not be admitted but show some promise, including throttling-down the acceptance of over-represented groups. This is a fact.

"predictive performant instruments" sure sounds very rigorous and objective, but you're talking about young people who are undergoing a lot of change. They're moving targets and predicting future performance based on one test only goes so far. A test like the SAT is always going to be a blunt instrument as is any singular factor. Admissions officers recognize this.

>or can pay full tuition

Can you name some "better schools" whose admissions are not need-blind? My understanding is that this is pretty much universal among prestigious schools.

Yes, but my knowledge of this comes from having an admissions officer as a friend, so I am NOT going to name a few highly regarded elite schools which do this.

Universities will certainly not publicly announce that they admit some students strictly because of connections or funding. Yet it happens... children of prominent alumni gain admittance with relatively lackluster grades, some numbers of foreign students which are required to pay full tuition get admitted even though they're not quite up to par with the other students. There's always plausible deniability, they're not going to admit idiots and at the same time many stellar students get admitted with no regard to need.

Admissions is not a "pure" meritocracy, nor is it the opposite.

"What we should really be talking about is improving these tests and beefing up operational security."

We have a real-world instance of the DRM problem here. Once you've put a static exam physically into the hands of the students, it's compromised. The attackers have massively, massively more resources than the defenders. The attackers can hire a hundred students to each take one picture of one page, and put the test back together themselves. And there's no way that the proctors can prevent students sneaking one picture of one page nowadays.

(Also, I'm not saying that's an optimal attack. I just put about 5 seconds' thought into it. But it's a sufficient attack already, so the existence of even further clever attacks only makes my point stronger.)

Even presenting the problems one at a time on a screen (massively more expensive, btw, due to brand new hardware requirements, plus introduction of other attack vectors) would not be sufficient. Clever games played with contrast or whatever won't work because the test needs to be highly accessible, so you can't play games like the old copy protection booklets [1], either.

Basically, if the test is static, it's game over.

Not that students aren't fairly good at cheating with dynamic questions. But those really only work well with the math questions, and even the dumbed-down calculators they still sell today can be convinced to run programs sufficient to "crack" any dynamic problems that the test companies can get their hands on for the math portions.

It's basically hopeless. The adversaries have gotten too darned powerful. They're armed with too much gear that would make old-school spies weep tears of joy.

[1]: http://i.imgur.com/eU8Rvhh.jpg

Might be true, still trying to measure the academic potential of a highschool student by a private company is absurd. I am all for participants doing their very best to "hack" the system. Kudos to those companies whelping them by take up the battle with the testing industry.

Wait for the higher ed crash. All things that can't go on forever, eventually stop. At that point the ACT/SAT problem will automatically solve itself.

The bottom line... with rampant cheating on standardized tests, fabricated school records, and massive "assistance" in college applications, it has become almost impossible to accurately measure the merit of most applicants from east asia.

First order effects arising from this are not that big a deal, those students will either wash out in in the university setting, or will end up cheating their way through school to the degree that they will be useless to many employers afterwards. The bigger issues is that, whether due to malice or not, schools have limited enrollment spots and people who try to "play fair" will be left at a disadvantage. Some might say that this is not all bad; that it prepares students for the real world. But I would like to think that we want to protect our culture/society from devolving into the sort of low trust free for all that exists in some other parts of the world. Anecdotally I've heard that this shift has already started to permeate our universities, where many students feel immense pressure to cheat because so many other people are doing it.

I'd be lying if I said that this didn't have a big effect on the way I view recent east asian immigrants in the workforce. Let you say I'm xenophobic, "recent east asian immigrant" would accurately describe many people in my family and social circles, and if I didn't open my mouth, the average bystander might assume I would be in that category as well.

The college board and individual colleges are not exactly innocent bystanders in all of this, and I'm glad I got through the system before these issues became acute, but I feel really badly for students from anywhere who actually play by the rules.

If a test is this easy to game, perhaps it's not a very good test.

Does Math and Language really correlate to ability? We know the answer is no. We sent humans into space, we can do better.

Has IBM/Google/etc attempted to plug Watson or any of the other advanced knowledge AIs into the SAT/ACT yet? I would be curious to know if they could consistently score a "high" score on tests without having previously seen the material (obviously excluding the essay portion).

Broken link.

As a society we should totally rethink how we handle children.

Employment is as personal and customizable as ever, but we increasingly treat kinds not even as blue-collar workers but as poultry on industrial farm.

Stick them in cage with artificial light, force feed with unnatural excuse for knowledge, disregard their personality, rank on how good they pass tests (we weight poultry at this point)

There's better education if you can afford it. There are schools where doctorates lecture the promising young on cryptography. There are schools with great faculty:student ratios. And great psychological resources.

Owing to how few of such environments society can sustainably create, I think it would be implausible to suggest that all children warrant access to such limited resources. At the very least, it would call for a rethinking of what % of our employable society should be employed as teachers, and how and where our most elite teachers ought be allocated.

I don't see why such environments got to be scarce. What's the limiting resource here?

The limiting factor is talent, bureaucratic and pedagogical, and money.

Teachers. Good teachers are rare and it's hard to measure teaching talent. Bad teachers are commonplace and unionized to teeth.

Sounds like a perfect environment for a disruptive AI tech. Like Coursera, but better and fully automated.

Or, leaving AI aside, at least another "Uber for X" app, where X = teaching.

Innovation and experimentation aren't welcome in education.

I recently wrote about this http://atom-morgan.github.io/child-labor-laws-are-outdated

I'm fairly young (25) and I agree 100%. I think I was treated like a child for far too long in an environment that doesn't make any sense. From the age of 5 until I was 22-23 I was sitting in a room studying the same shit over and over.

I'm in the same boat. It never hit me until my last couple years at university. I had professors teaching obsolete material who mandated attendance to fill out missing words on a PowerPoint that they purposefully left blank. I realized I was paying large sums of money to have my education actively inhibited.

>The traditional classroom is an obsolete detention home, a feudal dungeon.

-Marshall McLuhan

Reply on news.ycombinator.com