That's a good analogy. It's really hard to "measure knowledge," and in my humble opinion, as someone who once was a teacher, tests are one of the hardest ways of doing so.
I've never understood how someone could master a subject and yet be unable to answer any questions about it.
In my experience, people who did well on tests tended to understand the topic, and the people who didn't do well made excuses.
In classes where I knew the subject well, I generally did well on the tests. In classes where I had gaps in my understanding, I usually did poorly. In classes where the grades were posted publicly, my general subjective judgment of how well people knew the material matched up with their scores. Not perfectly, obviously, but the correlation has been high enough that I’ve never really been convinced that testing in general is “missing” some critical element of learning.
It would be a lot more acceptable if deviation of test scores from skill were to be totally random for every test.
These all affect how well you test, but do not determine your skill in the subject. Hence, having a disadvantage in any of the above gives you a system disadvantage in the educational system.
For one, the latter are often bullshit, as TFA points out. For two, they measure all sorts of stuff aside from actual proficiency. If nothing else, unless they took all the same classes - literally the same classes - throughout their school careers to date, no two kids got the same education. As anyone who's got the vaguest training in science can tell you, that kind of uncontrolled variability in your population will destroy any validity your measure might have. And lastly, so often these standardized tests really do have hacky questions put together with hack procedures. For my part, I distinctly remember completely stumping an IQ test proctor when I was a kid (yeah, I had helicopter parents). I had 5 cards, each showing a house, with the sun and shadows in different positions. I was supposed to put them in the correct order. So, naturally, my first move was to ask if the pictures showed the north or south side of the house.
There's no standardized test that measures the kinds of reasoning skills that really matter in life, such as the kind you'd use to make an educated guess that a question is being asked by the kind of people who would assume, unwittingly, that east is always on the right hand side.
I would expect the intelligent test taker to understand the test was not a trick, and that unstated assumptions mean use the defaults.
As for math, there are many ways to teach math, but 2+2=4 in all of them. It is reasonable to assume it is base 10, not base 3, unless the test said "in base 3".
Also good tests are hard to write. I’ve seen T/F questions that could go either way. Multiple choice questions with more than one correct answer. (Professors will tell you to choose the “best” answer. But that’s a matter of opinion in many, if not all, cases.)
I think what someone can DO with their knowledge is more important than what individual bits of knowledge they maintained. I’d rather hire or work with someone who can get things done and knows certain algorithms exist than hire or work with someone who can’t get anything done but can recite the same algorithms from memory. Tests favor the second person. (I was that person in HS calculus. Aced the class without understanding a thing just because I have a gift for remembering and applying rules. I had no idea what I was doing.)
This is why I am so glad that all of my university exams (and the _vast_ majority of exams before that, at least post-Y9/age 13) were open-ended questions[1], then marked by someone who will (likely) know the subject better than you even will. Even if you couldn't get the the answer, but could understand and articulate the starting points, or made a compelling argument but misread or misunderstood part of the question, you will at least get partial credit. The physics exams would also have a standardised formulae reference sheet.
[1] A typical paper would be three hours, answering 5-6 questions, and a typical subquestion can be as open-ended as "Write brief notes about a tree representation of functional arrays, subscripted by positive integers according to their representation in binary notation. How efficient are the lookup and update operations?"
E.g. Say I forgot the formula for Simpson's Rule on a test, but remember that it had to do with approximating integrals with trapezoids. Someone who thoroughly understands the course material could re-derive this formula in under 5 minutes if they had forgotten.
Right. And if you know the material, this isn't a problem.
How about spoken English? I know a great many people who are able to express themselves clearly and gramatically - surely mastery of spoken English - but ask them what the rule is for order of adjectives or some such and many of them would even need to double check what an adjective is.
The question of this nature about the English language would be "how do I know what order to put adjectives in?" and the answer would be something along the lines of "opinion-size-age-shape-colour-origin-material-purpose".
I do think they would be unable to answer. The rule of course then goes deeper, into ablaut reduplication and so on. People master spoken English without knowing anything [0] about it.
[0] Yes, I know, I said "anything" and now your literalist side is jumping in joy at being able to say "Aha, aha, they DO know at least one thing and therefore your statement is not literally correct!" I don't care.
Except, they can answer questions to show what they know, they just can't answer your carefully selected subset of questions with the exact answers you demand using the terminology you demand.
Draw an arbitrary line around the allowable questions and you can make every group of experts, "people who can't answer questions about it". But what use is that?
And it might be safe to assume a high test score is repeatable.
The challenge is to find a test that correlate high scores with good knowledge - not merely with being good at taking tests.
Because rarely do we care about how good someone is at taking tests; we'd like to measure how good their knowledge is. That's hard to do if we only can infer poor knowledge from a poor test result, but not good knowledge from a good test result.
Particularly with multiple-choice tests, this is not only possible, but in my experience common. It became a kind of running joke among my classmates that we know our teachers, not the material - over time we just learned the quirks of whoever designed the test and eliminated the wrong answers based on that.
Why waste your time learning that? Just learn the material. I'm often bemused by the effort people put into avoiding learning the material, much more than it would take to just learn the material.
He laughed, and said the stupid kids needed a break.
Still learning how to do "Professor X's questions" was a guaranteed ace. Forget the subject, memorize the borderline cases, max grade. Do all exercises on the text book, also, get a second text book and do all of those too, and you might fail the test.
In the end, tests end up approving a lot of people that learned next to nothing, as I myself succeeded in several by memorizing for the short term. I couldn't tell how many jingles I used to go through tests. In rarer cases, they also bombing someone with a reasonable understanding of the subject that maybe just was having a bad day or was sick.
And the idea of doing so is a fairly recent (and I would say toxic) invention. The concept of graded tests, marks and class grades (A, B, C etc) are all only a bit more than 100 years old. They're a product of the industrialisation of schooling.
I agree that currently, education systems are terrible, but is there any way to maintain them at scale that is not?
https://excellentjourney.net/2015/03/04/art-fear-the-ceramic...