The End of Scantron Tests | Hacker News Reader

The End of Scantron Tests

theatlantic.com

48 pointsgnicholas68 comments

I remember working on a project to put scantron machines in every public school in DC back in the 80s. We built the interface from the scantron machine to the DECmate II (a micro-PDP-8, if I remember correctly); async io in assembler... I learned a lot on that part of the project. Then we wrote the scanning software to allow lots of teachers to scan their tests in at the end of the day. Next we built a network over dial-up phone lines to allow the DECmates to upload their daily scans to a VAX (using Kermit, I think). Finally we built the tools to load all of the daily scans into a database and do all kinds of analysis and reporting. All pre-internet -- good times!

I remember learning a lot about the scantron forms and realized that if you made a black box at a certain place, that form would be interpreted as the answer key and would screw up a whole pile of scanning!

> if you made a black box at a certain place, that form would be interpreted as the answer key and would screw up a whole pile of scanning

Out of curiosity, where?

In the left margin, about 2/3 of the way down... between a couple of alignment marks. But that was 35 years ago... I could be wrong. I always felt like I held an immense power with that knowledge, but never used it!!!

The scantron forms that were in use while I was in high school had a sort of meta-data section, where students would put their name, the date, etc. One of those boxes was labeled 'key'. I always assumed it performed that function, but I never tried it to see.

Maybe the intention was to make it easier for teachers to identify a student maliciousmy marking it?

My recollection is that the "key" there allowed a teacher to give out multiple tests with questions in different orders, and then you would mark the letter for the test that you were given in that section. So test A would have questions in order 1 2 3 4 5, while B would be 2 5 4 3 1 and C would be 3 4 1 5 2. They could also change the letter of the answer, or really whatever you imagine.

Presumably it reduced people copying answers from their neighbors.

Tracks with my extensive experience of scantron in the DC public school system.

The worst thing about Scantron tests was when you skipped a problem and planned to come back later, but then forgot to leave it blank on the answer sheet. Then all of your answers were off by one, and you had to go back and erase/refill.

Some teachers would let you finish with the error intact, and then spend extra time (without the questions, just the answer sheet) shifting your answers down. But if it was a standardized test, no such luck!

> Soon, however, teachers and government officials sought ways to evaluate rapidly increasing numbers of students. In 1900, roughly 10 percent of teens attended high school; by 1940, some 70 percent did.

It reminds me of looking at a 1930s census, not long ago, while compiling a family history. Some of the adults on a rural farm were listed as having completed "4th grade". It's rather difficult to imagine a world where that was the norm and to compare how different our world now must be, when nearly everyone has completed high school.

I wonder how much the actual average level of education has changed, though the topics may be different. Could those 4th-grade dropouts read and understand a newspaper, calculate how many pounds of barley it takes to plant the south field now that we cleared that new area, and work out a 1-in-10 slope for the new shed roof? I bet most of them could, and that a lot of the time the average person spends in school now is not as well-directed toward their likely life needs — which may be less easily ascertained.

> a lot of the time the average person spends in school now is not as well-directed toward their likely life needs

When most people lived on farms (up until the 1950s), there was less need to be good factory workers. But there were more factories offering more employment in America, so the stats say you are correct.

https://qz.com/1314814/universal-education-was-first-promote...

What did they receive for humanities though? School is and should be about you making a better more complete member of society, not just a better cog in the economy.

> i bet most of them could, and that a lot of the time the average person spends in school now is not as well-directed toward their likely life needs — which may be less easily ascertained.

Interesting to order it this way because, as the engine of meritocracy, education is supposed to determine life direction, not be reflective of it. Of course this isn't how people actually think of it, but we don't live in a meritocracy either now do we

And pretty much all those jobs you're talking about are gone or low paying. Farmers aren't calculating how many pounds of barley per acre they need. They are operating computer controlled heavy equipment that does a huge amount of the calculations for them because someone thousands of miles away wrote magicial electric symbols in sand. The world is a vastly more complicated place now.

> operating computer controlled heavy equipment that does a huge amount of the calculations for them

They're also measuring soil nutrient and nitrogen content, integrating global price signals into their sourcing and supply chain, and paying attention to the genetic characteristics of their cultivars. I'd wager American farmers, today, are vastly more knowledgeable about their craft, at a fundamental level, than farmers in the 1930s.

Wow.

Someone doesn't spend time in rural areas.

Where I live it's almost all third generation "family farms", those that are still here after others have left.

The farms are bigger but they're still at core family farms and businesses with capital costs in the millions to tens of millions, run and worked by people that farm and almost all whom have other jobs and|or businesses in parallel.

One typical neighbour is planting out several hundred acres, has a side supply business with the grain coop to site several five story concrete grain silos, owns and runs four or five local school buses (and employs drivers) with a partnership in a bee hive placement business (for honey but mainly for pollination in the district).

Between them they all have a basic grasp of (with different members specialising) building, radio equipment, GPS equipment and data managment, double entry bookkeeping and employee paperwork for a million+ per annum turn over, mechanics for cars, tractors, bob cats, etc, agonomy, animal health, first aid, welding, carpentry, . . .

Hmm, while I think of it the same family has the local volunteer fire chief (another part time job) who handles bush fire preparedness in the area, fire breaks etc.

There's no reliance on "magical someones" thousands of miles away - those services are used, sure - but they're not counted on to be always available or there when needed - farming would just grind to halt with that level of unquestioning dependance.

>Someone doesn't spend time in rural areas.

Pretty poor assumption being that I come from a family of midwestern farmers that have been integrating high technology into their operations for decades.

>There's no reliance on "magical someones" thousands of miles away

You just listed out a huge number of things before that where they do have said reliance. Those bobcats/cars/tractors are not made in their neighborhood. Instead there is a vast network that supplies these objects to them at a national and worldwide level. If the fertilizer doesn't show up in spring, yields plummet. If the fuel trucks supplying petro stop farming in the mid-west stops. We aren't running steam engines that can be fed off of locally sourced wood (ok, maybe the Amish still are).

You live in a global economy that farming depends on. This entire rugged individualism works ok with short term problems, and falls apart as long term problems build up.

Take a look at this entrance exam to Harvard from 1869: http://graphics8.nytimes.com/packages/pdf/education/harvarde...

Good luck finding high school students today who could get a good score on this test.

There's vastly more emphasis on classical knowledge: Greek, Latin, ancient history, mathematical proofs...

If Greek or Latin we're still requirements for Harvard high school students would become fluent.

They sure would enjoy the salutatory speeches delivered at commencement: https://youtu.be/LYewkFKPPhs?si=egbRqSO2lMVxIMzz

I can't even figure out what some of these are asking.

Question 5 of the History and Geography section is just "Leonidas, Pausanias, Lysander."

Am I supposed to write about what they have in common? Who they each were? When and where they ruled? Other questions seem to include some form of instruction while being equally short ("Compare Athens with Sparta") but that one and a couple others are just a topic with seemingly no direction at all.

Heh, obvious a cultured man of knowledge (or anyone who watched the movie 300 and read wikipedia) would know that these are major leaders of Sparta

Grant, Sherman, Meade

Churchill, Roosevelt, De Gaulle

It's to filter the cultured elite from the unread plebes.

I would expect most Harvard-bound high school seniors to be able to breeze through the math section. Geometry was a ninth grade subject for me so I probably would have forgotten some bits if I took this unprepared as a senior, but it wouldn't have taken much studying ro refresh on that. There may be some trick question I'm not noticing, but the Latin section seems extremely simple. Ancient Greek has certainly vanished from HS curriculums, but if Harvard was testing on it two years of Latin and two years of Greek would suffice to pass those sections.

I think overall a Harvard-bound student today would do better on this test than a Harvard student from 1869 would do on the SAT, but I'm not sure if that even means anything.

The examination reflects the curriculum, and the preparatory schools taught to the examination.

Henry Adams on the Harvard of his day (he entered in 1854):

"disappointment apart, Harvard College was probably less hurtful than any other university then in existence. It taught little, and that little ill, but it left the mind open, free from bias, ignorant of facts, but docile. The graduate had few strong prejudices. He knew little, but his mind remained supple, ready to receive knowledge."

"In the one branch he most needed--mathematics--barring the few first scholars, failure was so nearly universal that no attempt at grading could have had value, and whether he stood fortieth or ninetieth must have been an accident or the personal favor of the professor. Here his education failed lamentably. At best he could never have been a mathematician; at worst he would never have cared to be one; but he needed to read mathematics, like any other universal language, and he never reached the alphabet."

But a) perhaps the mathematical curriculum had improved over the fifteen years between his entrance and that exam, and b) a large proportion of those who write about their schooling speak so poorly of it that one ends by suspecting exaggeration.

[Edit: in the spirit of "show your work": https://gutenberg.org/cache/epub/2044/pg2044-images.html]

That's an amazing test, if what you want to evaluate is the candidate's immersion in the wellspring of Western culture (and some math). I'm struck by how little fluency the Greek test requires, though: it looks to me (with my "small Latine and lesse Greeke") like it's mostly grammar work, since the difficult words are all supplied.

I would have loved living among people whose elites valued Western culture so highly.

You still can. You just need to play online matches of Rome Total War, which is where I learned the vast majority of my classical historical knowledge, lol

“It's rather difficult to imagine a world where that was the norm and to compare how different our world now must be, when nearly everyone has completed high school.”

Why? Are you sure that there was a significant difference between 4th grade in the 1930’s and 12th grade in modern days? Many students graduate high school barely able to read and write.

Surely you’re joking that you don’t think there’s much of a practical difference between an era where 90% of the population had less than eight years of schooling, and one where the overwhelming majority have twelve to sixteen.

I mean, you can literally just look up the statistics on literacy rates.

> I mean, you can literally just look up the statistics on literacy rates.

From what I remember they're shockingly low for a world where everyone finishes high school. IIRC only about 2/3 have even basic functional literacy?

You remember incorrectly.

https://nces.ed.gov/naal/lit_history.asp

https://en.m.wikipedia.org/wiki/Literacy_in_the_United_State...

Just because someone went to school longer, doesn’t mean that they learned more or learned skills they need to function in society. Also, schools today are full of distractions, class changes, unnecessary technology, and much larger class sizes than was present in the 1930’s. It's just not the same environment and simply comparing years of attendance is a naive approach.

> The SAT will go fully digital next year; the ACT, AP exams, and numerous state tests have already done so or will follow.

Future headline: teen hacker caught with stolen tests

Also might be a good attack vector for nation states. Could get assets into prestigious schools or publish anonymously online to cast doubt on the entire system

I've looked at the CollegeBoard's engineering job listings before and they were far below market rates. Wonder if they're adequately investing in their system's security

that's why credit score/agency and other finance sector companies have so many data leaks. not paying top dollar for top talent and just not a high priority.

There is a history of tests being leaked in Korea and China. Here is an example https://www.reuters.com/investigates/special-report/college-...

I followed the stories closely and what's weird is how little the College Board did about it. A number of times, whistleblowers would inform them about a test leak and even show them copies of it and they would do nothing about it. I think you're on to something as far as attempting to hack the SAT and the College Board just not having an adequate response.

Students managing to manipulate electronic school records has been an issue for as long as they've existed. There was even a well-known case as far back as 1986, which was caught on film [1] at the time and caused quite a stir.

[1] https://youtu.be/Hh_vLKlz2Mc

> I've looked at the CollegeBoard's engineering job listings before and they were far below market rates.

I assume you can make that up by leaking the tests.

I'm nostalgic for filling in those bubble sheets, even when I hated the tests themselves! They were a huge part of school culture.

In HS I bought a mechanical pencil with .9mm lead. I could bubble in the circles with just over one rotation, and the rectangles with a single swipe. It was the best.

Was it a Sumogrip, by chance? I had one of those too and it was awesome. I used it for everything and eventually lost it in undergrad.

Nope, but it did have a huge jelly grip like that. It was a super heavy metal pencil, which gave it a feeling of gravitas.

We are still forcing students to use the same shitty Texas Instruments calculators with horrible screens, unacceptably convoluted interfaces, and criminally slow performance and power consumption due to these same standards bodies. Technology has been better than that for more than 20 years now. We’ve gotten to the point where current students can use the calculator their grandparent bought as a child. How will moving the SAT from a bubble sheet to a laptop change any of the real factors that contribute to this problem?

I've never thought about the power consumption of TI calculators. I had an 85 and I recall changing the batteries occasionally, but not very often. I also didn't find the performance speed to be a hindrance during HS, including on the SATs.

What did you use your calculator for that it was noticeably too slow for? How often did you have to replace batteries? I get that there are newer technologies out there, and perhaps it would be a genius startup idea to make a device that's just like a TI, but using modern chips.

With current technology, 4 AA batteries should be able to power a calculator for an entire educational career from 6th to 12th grade, maybe more.

Edit: in terms of performance, the buttons and LCD are very unresponsive, you can type faster than they display. When making any kinds of graphs with more than 2 functions, the drawing happens in the order of 5-10 seconds, modern hardware could do that in micro-to-milliseconds and get the CPU back to sleep to save battery. Anything involving regressions of inputted tables can take 10s of seconds, which again feels clunky and eats battery.

No doubt. But is it a big deal to have to swap batteries once every couple years? I think the bigger waste is that these things still cost so much. I cannot believe that the TI-85 I used over 20 years ago is $70 on Amazon.

I agree, I think my point got bogged down in the details. Declaring scantron dead because the SAT is planning to go digital soon is so clearly out of touch nonsense on its face given how slowly technology is adopted in this space.

I dunno, my elementary school kids have never done a multiple choice test on paper. It does require that the schools have enough computers, but these days most schools seem to.

I always liked the sluggishness.

It made me think every operation through before I did it, so I couldn't just bruteforce results through guesswork. And the graph-drawing process was hypnotic.

For some models, you couldn't even cancel drawing once it started so you really had to validate your own inputs.

Everything about it forced an abundance of caution and efficiency...two qualities overlooked by so many "engineers" rushing through everything they do today.

I mean there are technologies that I think kids should learn that TI can’t even approach. Symbolic integration, for example.

But would a powerful calculator like that be allowed on the SAT?

Why shouldn’t it be?

Well, some TI's aren't even allowed on the SAT. I think the reasoning is that they can solve equations for you. They do allow other TI's that can do this, but only if you format the equation appropriately. If GP is envisioning an even more powerful calculator, then it certainly wouldn't be allowed.

I get that it wouldn’t be allowed on the extant SAT, I am saying the SAT should be redesigned to include higher level questions that allow the use of technology like symbolic manipulators

As I struggled about 6 years ago to work the TI graphing calculator that was necessary for my Calculus course, I found that there were alternatives. There are emulators that you can just install and run on Windows. Even crazier, I was sorely tempted to use those websites where you could just enter in a formula, and it would do the derivation or integration totally automatically. I mean, I knew I'd get answers that way, but not learn the process. It was one of the most difficult classes for me. I believe that the IT-related majors later abolished all of their Calculus requirements.

Have you seen the newer calculators? TI does still sell the older calculators but the newer ones have color screens at least. The interface is different (from my 2000s TI-83+ and TI-89) too.

Also, I would have appreciated the app/online version when I was in school. The iPad app looks really nice. I felt a bit of jealousy when I saw it lol. I know you mentioned College Board’s calculator rules and iPads obviously wouldn’t apply but you were also speaking about the calculators in general.

But are the calculators actually “shitty”? I have fond memories of my TI-83 and TI-89. They worked well. I don’t recall being annoyed by anything. But also I don’t tend to complain about things like this. Never have. For example, is a high schooler really complaining that much about battery consumption for their calculator? I don’t recall this at all in middle school or high school.

I can see though how the general student could benefit from something slightly better. I can also see how these calculators would feel outdated to a current high school student. I just don’t think they were or are that bad lol. But again, I’m not one to overexaggerate things in the first place lol.

It would have been lousy if the calculator just stopped working without notice. But my recollection is that you could tell when it was getting low on batteries because the display would get faint. Ah, the memories...

Yeah, those things had 9 levels of contrast. With a fresh set of batteries the display would be crisp and contrasty at “3” or something. Then slowly you needed more and more levels to feel normal. Before you know it you’re running at contrast “9” and contemplating your next score. Then you realize, “oh yeah, batteries are on their way out.”

As you snap the final AA place, you sign contentedly as you dial it back down to “3”.

Hmm. I’m not bragging at all but as someone who got a perfect score on the math section of the SAT (since scantron/bubble answers are being talked about) and enjoyed math a lot, I don’t recall this at all lol. Do you mind mentioning the year(s)? I said early 2000s in my first comment. The batteries lasted, at least for me. I recall having to replace them for sure. But it was never a big deal. And I’m speaking as a Black, lower middle class person growing up. It wasn’t that serious to me or the other people with similar backgrounds at my school(s). But still, I can understand why/how it would be serious to others (as I mentioned in my original comment).

I had a TI-81 in 1993, and upgraded to a TI-85 in 1996 I think. I didn't mean my comment to be an indictment of TI calculator battery life. My experience with them matches yours. They were great!

I do remember the ominous feeling every time I noticed the screen getting fainter, checking to see what contrast level I was currently at. But it wasn't actually that big of a deal.

I probably only needed to change them twice a year or something.

> I’m not bragging at all but as someone who got a perfect score on the math section of the SAT

This is very strange behavior for HN (especially because many HNers got 800 on their math SATs, I would imagine).

No. My goal was to provide evidence/an anecdote to back my thoughts/opinion about these calculators. I didn’t do any of the math competitions like AMC or olympiads or whatever like people on HN talk about on occasion. This would have probably sounded more appropriate: “As someone who did AMC and was really interested in math and used those calculators a lot, …”

But I couldn’t say that. So I came up with the closest thing I could have at 5AM after a night of partying. I did notice that it might sound a little douchey which is why I added the “I’m not bragging” part.

But thanks for assuming the worst! I believe that’s against HN’s guidelines. What I said is not much different from saying “As someone with X years of experience in Y engineering field, I think that…” People say things like that all of the time on HN, right? Many of us have similar X years of experience in Y engineering field. But we still say it.

> (especially because many HNers got 800 on their math SATs, I would imagine).

For sure.

I guess I was also shocked that people who have similar math backgrounds would be complaining the way you were about these calculators, again given my fondness for them. In my first comment, I did also mention that I don’t tend to exaggerate. So maybe that’s partially the disconnect here?

The other part was that I was pretty sure that you guys were talking about calculators from the 80s or 90s. I asked that question also but no one responded at that time. People just kept complaining about problems/issues with the calculators lol. (Someone finally replied and mentioned 1993!)

Sorry, but you guys were starting to turn this into some sort of Texas Instruments bashing circlejerk, which didn’t seem fair to the product.

Sidenote: HN isn’t all fun and games. Please don’t paint it as that. When I (and peers that I’ve spoken with) talk about HN, it’s usually characterized as a forum where really smart people like to bump heads. A place where people like to outsmart each other. A place where people nitpick article titles. You get the idea.

Your behavior correcting skills were needed during plenty of more appropriate times. For example, when someone with a username like “koreanguy” said that he wanted to personally kill Timnit Gebru if he could. No downvotes or flag after many refreshes. Want to know why? Because plenty of other men were leaving similar disgusting comments in that post. This is just one example out of many of “weird” behavior on this forum. How many times has the mod here left that “be nice and follow the guidelines” comment on charged articles? (Some) HN users love a good dogwhistle!

But my little SAT comment is deserving of a reprimand lol?

Hilarious to quote the guidelines at me. I wasn't "assuming the worst" — I was just pointing out that it's very odd to gratuitously mention your SAT scores. (I've been on HN for a decade and never seen anything like it; it looks like you're rather new, so might not have that background.) It doesn't prove how much or little you used your calculator. People who get 650 or 710 use their calculators too, perhaps even more! If you had said "I used my calculator a lot, though not at the level of math competitions," that would have been more enlightening. I appreciate your mentioning the 5 AM partying context, which definitely helps.

I, too, have been using HN for over a decade. I never add my email to the accounts for recovery lol.

> perhaps even more!

And that’s also part of the disconnect here. I don’t believe this to be true fully. Growing up, the people who performed the best were the ones that really delved into the features of the calculator. They also installed more games. Did programming. The ones that did more than what the teacher taught. They would be the ones to remember whether not there were “devastating?!?!” contrast or battery issues. That’s just my opinion and experience (which is also why I provided other relevant personal details for perspective).

I do understand your line of thinking though (e.g. people who perform the worst use the calculator more because it takes them longer to do homework). The question then becomes this. Who has the better, unbiased memory of how the calculator functions: the struggling student or the one who cultivated a fondness and really explored its capabilities?

I did mention that I used my calculator(s) a lot in the same sentence that you had an issue with at least lol

Also, I just searched for HN comments related to SAT. There are a few from high schoolers and adults. Not many. But enough to satisfy me lol!

It's also worth considering their competitors. My experience in the UK system was that TIs were a rarity and most people used graphing calculators from Casio (if they felt the need for an upgrade from a scientific calculator at all), which were cheaper, lighter, widely available in any good stationery section, and appeared more feature-rich (better screens with intuitive menus, built-in SD card slots, and probably better processors). In particular some TIs I saw lacked graphical equation display, which would be a deal breaker, exotically when our existing £15 scientific calculators had made it a basic feature

There was no requirement from the schools or exam board to use a particular brand. In theory you could do everything you needed with a basic scientific calc like the ubiquitous FX-83GT (and many people did), and the only real rule was "no computer algebra systems" (I believe the TI-89 was given as an example of a disallowed calculator). In fact the teachers were somewhat discouraging of graphing calculators at all, saying you can get one if you feel the graphs help you, but you shouldn't need anything more than what you had in GCSE

In the US, students were allowed to use the TI-89, but not the TI-9X, which. Had a QWERTY keyboard but was otherwise identical. The 89 could solve equations, but the 85 and 86 could do the same, if you inputted them in standard form (y = mx + b). It could also solve simultaneous equations, again if you standardized the formatting. This was definitely a useful feature, beyond what scientific calculators could do.

In the early 80s our entire SAT prep session consisted of being told one high school morning: 1 - fill the bubble in completely with the right kind of pencil or the answer wouldn't be seen by the machine and 2 - the formula of "if you don't know the answer, if you can eliminate at least one option, it's worth guessing".

A lot cheaper and more time efficient than buying a book, much less taking a prep class. Perhaps those options didn't exist back then?

"That model was more holistic than a multiple-choice test, but also prone to subjectivity and bias—and only possible, in part, because far fewer children received a formal education."

"If the epoch of multiple-choice tests is truly ending, the assessments won’t necessarily be missed. Not only is the format inherently reductive—bubble-in question-and-answer forms have also been prone to bias."

So all formats are biased.

> So all formats are biased.

Bias isn't a binary quality, so that's an oversimplification.

Misleading title. Scantron tests have not ended. it's about proposed alternatives.

Reply on news.ycombinator.com