Hacker News Reader

Comment by jonplackett | Hacker News Reader

The problem was not using an algorithm, it was that they used a crap algorithm that could award students grades that were higher than those even available on the paper that they sat, and made it impossible to get a high grade if no-one fromm your college got one before.

I'd love to see the details of the algorithm itself. Surely it can't have been too difficult to put some guard rails in there to prevent it moving any grade more than say, two places (maybe even one) from the predicted grade.

Instead it seemed to have the freedom to do whatever it liked. Moving Bs to fails. Students predicted to fail getting As.

That's just incompetence on the part of whoever made that algorithm.

View on HN

And don't forget the totally illogical situation of awarding someone a higher grade for advanced maths then they got for basic maths.

I had a passing thought of a different system - using basically the same calculation of a school's predicted grades, allocate each school a 'budget' of marks they can distribute as they see fit, based on whatever criteria they pick. Just to be clear, I don't think that's a particularly good system either, but a canny government could have deployed it to avoid much of the direct criticism.

So then you'd get parents complaining about teachers picking their favourites and downgrading pupils for personal reasons, which still would have been the government's fault for implementing a system open to such abuse.

Like all algorithms it had bias built in - one of which was to inflate private school grades, making grades even worse of a measure of quality than normal.

What was particularly inexcusable though was the government response - the implications of the algorithm were available to the government weeks before the results, and the public backlash from Scotland having similar problems happened a day before.

Now you could argue that to avoid grade inflation as a while the government should have proceeded regardless, however wen the inevitable backlash occurred (predictable since July and known since the Scottish exam results earlier in the month) they turned round.

This u-turn came too late for many students to benefit yet wiped out all benefits of grade inflation prevention - literally the worst of both worlds.

A description of the algorithm and its shortcomings:

http://thaines.com/post/alevels2020

and made it impossible to get a high grade if no-one fromm your college got one before

Well that is unlikely. If you are going to handle it algorithmically then what else would you expect. Clearly the teachers can't be relied on because now everybody is getting an A*, the grades are effectively useless as an indicator.

I was disappointed although not surprised that the government caved on this.

Personally I’d expect assessment of an individual to be about their performance not their schools past performance. Any system not predicated on that is doomed to fail in assessing students performance as this one obviously did.

Well that's what exams are for but they went out the window as soon as the virus arrived. Whatever system the govenment put in place would be worse but it didn't need to be this bad.

They could and probably should have done exactly what ended up happening and relying on prior data per student. Imperfect though it was at least it's about their attainment rather than an aggregate attainment.

Two reasons: The results indicate that the algorithm was flawed: "Ofqual figures show 39.1% of 700,000 teacher assessments were lowered by at least one grade"

Secondly, many students and teachers complained. This second reason is the thing that "the world can learn" that the article focuses on.

Which can be explained by teachers assessments being too optimistic. Now we have reverted to that you can see the results, grade inflation.

Of course students complained, I would if I didn't get the grades I was hoping for. That doesn't mean the government has to cave in and pretend everybody is a winner.

We really need to see what teachers assessments in previous years look like. Are they consistently optimistic?

You don't have to look far, now the government has caved we can see the grade inflation, particularly from state schools.

That is showing this years assessments, which I imagine, were done during the lockdown. I'm wondering about previous years.

> Which can be explained by teachers assessments being too optimistic

This can also be explained by teachers assessments being an accurate representation of students ability, but students traditionally not faring as well in exams as their ability would suggest due to exam taking being an additional unrelated skill.

fwiw though, you only need to look at some of the outliers to see that the algorithm failed. It's all very well to say that it produced a consistent result in general, but each result specifically applies to an individual person so needs to be fair to each person as well as in general.

Whatever algorithm used though the truth is that qualifications achieved this year are not directly comparable to qualifications achieved in other years due to the lack of exams.

While nobody's fault, imho we'd have been much served by admitting that they're not comparable and doing something else (e.g. add an extra qualifier to the grade or something) than by trying to make them comparable and obviously and predictably failing at it.

> fwiw though, you only need to look at some of the outliers to see that the algorithm failed.

I'm not arguing it's perfect, I'm just saying what we have now is worse in the aggregate. If you inflate so many grades then the grades become meaningless. You have devalued the work of the high achieving students.

Not really, predicted grades have exam taking ability built in. The predicted grade is the assessment of how the student will perform in the exam. Coursework that was completed was already marked and the component of that set normally.

It says this is happening in the article.

It's not unlikely, it's practically certain. If you have hundreds of schools there absolutely will be a handful of students who do better than previous years and there will be a handful of students who do worse than ever before. Maybe it's not a big deal to hand a C to someone deserving of an E, but it's certainly a big deal that the moron at the school that gets good grades (read: private school) gets a pass, whilst the genius at the school that historically did badly (read: state school) gets screwed. Now, given that's how this system works, take a look: do you think the politicans repsonsible for this went to the state school?

There are always outliers.

Interestingly the private schools seem to be to be grading their students more accurately, the problems were more common in state schools. The moron in a private school was unlikely to get their grades inflated. I'm not a fan of private schooling but in this case they appeared to be doing the right thing.

There will always be outliers, no system is perfect but what we have just done is worse in my mind.

The algorithm was the same one used for decades, and previously published. It pretty much says:

> You get your exam grade. If that isn't available, you get a grade set by averaging your peers from the same school, with the set of peers decided by a ranking set by your teacher. If that isn't available, you get a grade chosen by your raking in class selected from the distribution of your schools past performance.

There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers. We already suffer an element of grade inflation, which causes employers to say things like "You must achieve grade A* in Maths to apply for this job", and applicants from years ago before that grade was even introduced are automatically excluded.

> There isn't really any fairer way that doesn't lead to grade inflation when there is an element of dishonesty/optimism on the part of teachers.

IMO it's much better just to swallow the grade inflation for a year. It will naturally reset itself next year (when exams will be sat again), and the negative effect from it are pretty minimal.

The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold.

> IMO it's much better just to swallow the grade inflation for a year

You're not alone, this is the thinking they're using. [0]

> the negative effect from it are pretty minimal

That doesn't sound right. Top-flight students will be disadvantaged.

[0] https://www.bbc.co.uk/news/uk-53810655

> IMO it's much better just to swallow the grade inflation for a year.

When hiring, I'll be crossing off any qualifications earned in 2020. As well as the results being 'fake', the actual learning was in most cases not being done because the students knew they could chill at home and not learn. Sure, some students were studying, but as an employer I don't want to take that risk.

Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams?

>Would you want your doctor to be the one who messed around on TikTok for the whole year of his final exams?

Well they still have six years of medical school to get through. Perhaps the dropout rate for that will be higher.

>The algorithm is based on the assumption that performance of successive school years at a given school is roughly consistent, not only as an average but as a distribution. And that assumption simply doesn't hold. //

It seems likely to hold pretty consistently.

What's the variation?

When did the algorithm get used in the past? I imagine it being applied to individuals when they had an accident and missed the exams, but did it get applied to entire schools?

It got applied to entire schools if for example the school had to close due to a fire/flood.

My school had it applied to one of my exam subjects for the entire year. I suspect the exam papers were lost/stole/destroyed after we took the exam, but before they were marked. Not sure how, but everyone was assigned 100%, and nobody complained...

There's more to it. The system being used will lead to grade-inflation, [0] as there are two grade-estimation systems in use and the student gets to keep the better of the two.

The two systems used are teachers' grade predictions, and the grade-prediction algorithm.

We discussed this two weeks ago at https://news.ycombinator.com/item?id=24191882

[0] https://www.bbc.co.uk/news/uk-53810655

The algorithm you describe seems to have nothing in common with what has happened. There are students getting many grade levels above or below what they were predicted. That makes zero sense.

Grade inflation is a completely different topic and has happened even more now that they've decided you can have whichever grade is higher out of predicted VS algorithm, grades are up something like 20% on last year.

Reply on news.ycombinator.com