Hacker News Reader

> If you didn't get 99th percentile the course was refunded or you could take it again

Plenty of other domains have study programs with similar guarantees where the program has little to no effect-- it's a simple economic calculation: Set your rates so you make a good profit even if the program has no effect (even if you allow retaking the course, students won't be willing to waste their time and yours forever).

It would be more informative if you knew the actual before/after performance for the program. Elsewhere this has been studied (see links in the thread) and the improvement wasn't that substantial.

> You can't retroactively improve your GPA by doing better in the past.

Indeed, which means that people who's families have been carefully shepherding their education since they were much younger have an _insurmountable_ advantage in the GPA game. For people people who don't come from highly educated families, they only learn about the GPA boosting games as they start thinking about college years to late to take full advantage of them.

>Plenty of other domains have study programs with similar guarantees where the program has little to no effect-- it's a simple economic calculation: Set your rates so you make a good profit even if the program has no effect (even if you allow retaking the course, students won't be willing to waste their time and yours forever).

They offer a refund or to retake the course. It is indeed a simple economic calculation. The company would quickly go bankrupt if a sizable percentage of students were refunding. The company itself, and the location I worked, had excellent reviews. Even today it has a 4.9 star rating on Google Maps. Quite a high rating for a service that people in this comment section proclaim has no impact.

Intuitively, how could it be correct that people pay thousands of dollars on these courses and receive zero or near to zero benefit from them? If this were true no one would take the courses, the courses would be rated poorly, and the underlying business would fail.

>Indeed, which means that people who's families have been carefully shepherding their education since they were much younger have an _insurmountable_ advantage in the GPA game.

Sure, but the context is "which is easier to game." To game the GPA takes four years of study and prep and you must start doing it at the beginning of grade 9. Tens of thousands of dollars and preparation+work over multiple years. To game the SAT or ACT you must spend a few thousand dollars during grade 11 or grade 12.

> To game the SAT or ACT you must spend a few thousand dollars during grade 11 or grade 12.

"Or check out a SAT book or two for free from the library.", says my partner from an extremely poor family who got into college on the basis of her perfect SAT score and whom never would have qualified to a prestigious school on a GPA basis. (And whom was also admitted to law school on the basis of a nearly perfect LSAT, which she studied for only with free and extremely low cost used materials)

> Intuitively, how could it be correct that people pay thousands of dollars on these courses

People, particularly those to whom a thousand dollars isn't a big deal, spend all kinds of money on speculative and outright ineffective treatments. Including mystical mumbojumbo, quack medical treatments, and products and services which accomplish nothing except contributing to the 'identity' they present to themselves and others. ("I am a parent who cares, look I spent $zillions getting Jr the best opportunities!").

> To game the GPA takes four years of study and prep and you must start doing it at the beginning of grade 9. Tens of thousands of dollars and preparation+work over multiple years. To game the SAT or ACT you must spend a few thousand dollars during grade 11 or grade 12.

I think on this point we're agreeing to a great extent but we're drawing opposite conclusions. I agree SAT improves with focused study, though I believe that improvement is available for free (other than time and knowing you should do it).

You seem to agree with me that it is very expensive to hyper-optimize GPAs, requiring costs and actions extremely early and on a sustained basis.

My conclusion from this is GPA optimization relatively more available to students with more affluent families, because it takes more time, more money, and requires it earlier and more speculatively. -- we don't have an option where you can't improve your performance with the input of time,money, care but we can choose metrics where the available improvement is available to more people.

> Intuitively, how could it be correct that people pay thousands of dollars on these courses and receive zero or near to zero benefit from them?

Because the people paying have a sample size of n=1?

Look, any test will be subject to random measurement noise. If you got a bad night's sleep, got lost on the way to the test, had an argument with your parents or broke up with your boy/girlfriend, you're probably going to do worse. Conversely, if you had a nutritious breakfast or take a version of the test with lots of questions similar to the ones you practiced on you'll do better. Some people get "lucky" (for lack of a better word), while some get "unlucky".

The thing is, unlucky people are far more likely to retake the test and pay for test prep. So people enrolled in you test prep class are NOT a random sample of high school students. The sample is biased to include more unlucky people and fewer lucky people.

People paying for your service don't know whether their child was unlucky or not.

I used the work "lucky" for a reason. It is less likely that those negative factors would affect the student both times they take the test than just one time. So someone who encountered a confluence of unlucky factor for the first test might score higher on the second test simply by having fewer unlucky factors--or even just being more familiar with the test, having taking it previously.

> If this were true no one would take the courses, the courses would be rated poorly, and the underlying business would fail.

Your course doesn't need to actually help anyone. Your customers just need to believe it helped them. How do they know whether you actually helped, or whether the improved test scores are down to luck?

Does your average customer have 20 grandchildren who they can randomly split into test and control groups? Or do they have an imperfect understanding of statistics?

Maybe there was another variable at play: language. SATs are in English.

If English is not a student's primary language and fluency improves as they advance academically, up to a plateau with age.

Not sure if this was properly controlled in SAT studies. https://www.ncbi.nlm.nih.gov/books/NBK11007/

Glad you mentioned this. I went to a strong wealthy public high school with a high asian population (~50%?). When I was there, a lot of my friends were 2nd generation immigrants and their parents still spoke their native language at home. Their kids (my friends) were perfectly fluent/native in English but didn't do as well in the "edge vocabulary" parts of the SAT and I always figured that was why (in comparison to me and my english-at-home parents).

> Candidly, no one would pay for a prep course for a 30 point gain.

My understanding of the research in this area is that the 30 point gain is if you compare people who take a prep course to people who put in an equivalent amount of time preparing on their own with the widely available (e.g. can check them out of the local public library in many cases) preparation materials.

The 200 point gain is if you compare people who did the prep to people who have not done any prep.

The set of people who can't make themselves put in the time without the structure of a course would benefit by a lot more than 30 points from the course, and would pay for it. And do.

I vaguely recall a question on the Duke University application about what assistance you received with your college application efforts. (I can’t remember if test prep was included specifically.) I’m sure that some people lie by omission, but maybe asking this question is better than not asking it.