Automatically grading multiple choice exams from photos using Python and OpenCV
pyimagesearch.com
pyimagesearch.com
The idea is that answers are highly correlated. Better students are more likely to get all answers correct, worse students are more likely to make mistakes. So if you do something like PCA on all the students' answers, the first dimension will represent the quality of the student. And the weights will represent which answers are correct.
In fact I believe there are similar but better algorithms than PCA for handling mutually exclusive binary variables. I just don't know what they are called.
I would love to see an automated "quiz creation" engine (perhaps with some minimal instructor curation) that would both quiz/score students - but also allow teachers to integrate more assessment, more often at lower stakes.
By significantly reducing the labor involved with assessment, teachers can more effectively create learning opportunities for students (teaching - with all it contains - is time constrained). Certainly, self-leveling assessments already exist and are in common use with cognitive tutors/online education; however, the knowledge/skills they assess must either be very general or specific to the content of the online course. If teachers were tagging the content of their courses - which are likely to teach knowledge/content that is being taught in hundreds/thousands of other school across the country - then such a self-generating assessment engine would be very useful if it could self-modify to accurate assess whatever is being taught in the course.
Modifying learning materials and tasks for different student levels is a major pinpoint for teachers, but is very much a stated goal in almost all k12 schools. It is done poorly, if ever - and mostly when legally required by IEP/504 plans. The trend is very much in the direction of increased data/personalization in the classroom, but currently the big investments in data are being made on the administrative side of the k12 industry. There is so much potential for improving the classroom backend to support instructor workflow rather than constraining it with pre-conceived notions of how we work/what our goals are. I hope to see it happen.
That is an absolutely fascinating idea. I would love to see someone run with this as a company, or as an internal initiative at the Department of Education- that seems like a brilliant idea. Has anyone done this before? Is there any reason why this isn't a slam dunk?
I would be also be intensely interested in exploring this as a domain side professional.
For this to work, you need to do three things: (1) get the original exam, (2) tag every question with a series of categories, and (3) get the students' completed exams. You'd think professors/teachers would be excited by the idea of reducing their workload with an automated quiz/exam generator based upon their curriculum. It didn't seem like it.
The professors can't be relied upon to tag their exam questions with categories, so the company has to handle #2. To solve problem #1, we reached out to hundreds of professors at multiple universities, explained what we were doing; and asked if they'd be willing to share their corpus of past (and hopefully present) exams. Nope! They didn't want to do it, and when pressed to give an explanation the lines of communication went cold. Not to mention, if we couldn't convince them to share a blank exam, there's no way they would be willing to share their students' completed exams.
So, we thought of a different route. My cofounder had taken a year off from Vanderbilt University, where he participated in a fraternity. Not only did his frat have a massive test bank that we could use, he knew plenty of other frats that had even more. It was a start. But convincing frats at other schools to upload their exams was a no-go. They slammed their big Greek doors in our face.
There was nowhere else to turn, except the student body itself. But no student wants to give away their exams without an incentive, and we didn't have the money to pay them. So, we setup a marketplace where students could upload their exams and sell them for a split of the sale. Finally, we got some traction. Some students uploaded their exams. Other students bought them. In our first week of release we made almost $1k. We were on track to finally make good on our promise; it looked like we'd be able to hire people to annotate the exams and begin work on our exam generation service.
Until that fateful night, while we sat there trailing the logs, and we saw it. A Vandy professor had signed up and bought one of her own exams! (Just to verify they were authentic.) She immediately reported my cofounder to the university's disciplinary department. And they threatened to expel him with a permanent mark if he didn't immediately take down the website. After 3 years of classes, he'd leave with no credits, no diploma, and no other place to go. Like most young kids, he decided to lay his cards down under such massive pressure.
After that, we shut the website down, and went our separate ways. Never able to achieve our ultimate vision.
Maybe there's a different way to go about it. But without a strategy to compile an extensive corpus of completed exams with tagged questions, I don't know how you would go about it. If you have any ideas, I'd love to hear them. Because not a day goes by where I don't want to take another stab at this.
Even in k12, there might be a chicken/egg situation. The obligation to tag content is a barrier (just as tagging is always a barrier).
That said, k12 has so many underlying content standards - which are only becoming more entrenched, for instance with standards based grading - that you might want to integrate tagging as a sub feature to a curriculum creation/curriculum performance toolset so teachers don't have to focus on it specifically (only, rather, as fulfilling the pre-existing expectation that they are hitting their standards).
But then you'd need a toolset teachers actually want to use.
Do you have any links to information on the current market for curriculum creation/curriculum performance toolsets? (Something with a list of companies, their products, total market penetration, and respective market share.)
There is a diversity of approaches to improving schools and education. They often feel as much like features than they do companies.
I have to feel that creating the content/performance/assessment backend opens up a lot of progressive improvement. You'd actually be able to see inside the classroom in a broad and detailed manner. You'd actually be able to personalize instruction without diminishing content to paint-by-numbers. And, maybe - eventually - you'd be able to de-school education and let people learn as they can, where they can.
I'm actually working on a very small (but lucrative) experiment on building such a de-schooler, in a very specific vertical, as proof that it can be done. Then, after proving the process, I planned to take the lessons to their logical extreme. If you'd like to talk more, please email me: denzel dot morris 1 at that google mail service.
One goal of creating such a system is to collect and make re-usable (and re-mixable) all curriculum materials, performance related interactions, and assessments that are being created and used everyday in our schools. Perhaps, US students won't take advantage of the fact that a few decades of this material might mean that personalized learning opportunities can be created for them without the intervention of a human being - but it could develop as a parallel capability underneath the current school system to support learners anywhere in the world who have non/minimally functional schools (or, for that matter - anyone who would prefer not to spend their days in a school). It might not perfectly rival a high quality flesh-and-blood instructor, but it sure might be better than the status quo for that person elsewhere.
It's the difference between [Sugara Mitra's hole in the wall] or [Negroponte's OLPC] and real education. Computers are wonderful - but most young learners need structure. Likewise, when Facebook/Google/? blanket the world in free internet we will see a flood of brand new educational products, but why create an entirely new army of curriculum developers when we already have an army of 3.1 million trained school teachers creating educational content every day? All of that labor is silo'ed within the individual classrooms, both preventing us from self-improving as well as providing educational opportunity broadly.
Also, the outcome would not be to close your neighborhood school. After all, community schools exist for a variety of reasons that are distinct from education. Schools won't go away just because education can be done without them (as that is already explicitly true).
Yes! Until you brainfuck them, by putting the correct answer in the column B for the first 10 questions. Once the doubt is there, it's not gonna go anywhere.
Note- I'm not challenging you- I don't see why it shouldn't be the case- I just don't follow your reasoning (but I'm quite interested in the claim).
Is it because the largest determinant in which answer is correct is the quality of the student?
Disclaimer: I bought his eBook bundle.
His book takes it to another level, but isn't necessary to get your feet wet.
It was easy for the students to double check that their tests were marked correctly as well.
[0] https://blogs.nvidia.com/blog/2016/09/02/gradescope-brings-a...