On the efficacy of online proctoring using Proctorio (2021) [pdf]
ris.utwente.nl
ris.utwente.nl
Whether the integrity of the exams is actually guaranteed is immaterial, because the company that made the nanny software went on the record and took responsibility, and/or by splitting the responsibility, the school admin and the nanny software vendor can point fingers at each other and the responsibility essentially disappears like it usually does.
You can see this with most other security theater type things. As long the "general public" is unaware that they have been lied to, they will accept this kind of positioning as gospel, and not look into it further. For the most part, as expected, the world turns around just fine without real security because systems tend to be robust to a certain level of bad actors doing selfish and small scale bad things like exam cheating, and as long as nobody's feelings are hurt, it's all fine.
Funny you mention this. I remember a senior staffer at a large games publisher talking about this back in the late 00's.
He said, quite candidly, that all of the execs at EA, Activision etc. all knew that technology like SecuROM was incredibly ineffective at reducing piracy.
They all paid millions of dollars in licence fees to use it, though, so that they when an investor asked them what they were doing to protect their IP, they had a "good" answer.
AACS was supposed to last years. HDCP, indefinitely. BD+, the creators thought it would last a decade at least. Cinavia? They thought it would never be removable. Drive Bus Encryption? Almost impossible they thought.
All of these technologies were defeated about ~2 years after their introduction. Some had minor updates breaking things for a little bit, but were broken again shortly later. And some are only partially broken (not enough to make your own player), but in practice broken enough where you can rip the disc which is all most people want.
However... even the best intentions went awry after the first Blu-ray player software for PC was released with the flaw of allowing Print Screen to capture frames. Which caused people to quickly figure out they could run a script using Print Screen to capture all the frames, while having their audio-out headphone jack connected with a male-male 3.5mm cable straight into Line In. No real hacks necessary!
This was, of course, patched by blacklisting the player key and rolling out new keys in updated player software (to render new discs unplayable without an update) but it was a major oversight. ;)
But they heaped on BD+ VMs, Cinavia watermarking, BD-ROM Mark, basically threw everything they had at the problem and hoped that something would stick. Movie studios actually say that Blu-ray supporting more DRM methods was actually a major reason they chose Blu-ray over HD DVD (which didn't support BD+, which supposedly was DRM designed to last a decade).
And again, if that perspective were true, the movie studios wouldn't have done the whole dance again with HDCP 2.2, BD+2, and AACSv2 when 4K Ultra HD Blu-ray came out (only for all of them to get broken again not long after release despite being completely-fresh implementations). You already needed the older versions, right? So, I would argue, it's actually a mixture of both.
It's funny, songs are much easier to share than movies, the files are a hundredth the size, yet the movie industry thinks DRM is very important while the music industry survives just fine without it.
OTOH, video is just pixels and waveforms. If you can get that raw content out somehow, copy protection cannot prevent someone from enjoying that copy (which can then be re-encoded to not have protection). So making video copy protection is much harder because you actually need to prevent copies rather than just make sure the authentication is hard to break.
There are actually a handful of technologies meant to survive the transcription process only to reappear in the new medium. Probably the most effective schemes are used in money, in paper bills. There are little things there (ie the eurion constellation) that are purposely designed to be scanned. They can be copied. But then they are detected by software (Adobe Photoshop) and hardware (high-end printers) which then refuse to edit or reprint them.
https://en.wikipedia.org/wiki/EURion_constellation
Other tech is meant to leave a trail leading back to forgers, a bit like a DVD ripping software embedding your MAC/IP information in any Mp3 it generates.
>Other tech is meant to leave a trail leading back to forgers, a bit like a DVD ripping software embedding your MAC/IP information in any Mp3 it generates.
It is simple: rip the media multiple times with different identifying information, and then analyze the diffs (or just take the average). You cannot hide the existence of steganographic data if there are multiple copies with different steganographic data. It is not like analog where a loss of precision is justifiable
As for fingerprinting, that's a whole 'nother can of worms. I've never seen it be used for DRM specifically but it has been successfully used many times to track down who was leaking trade secrets etc. In fact it might be more widely deployed than most people realize.
The problem with fingerprinting for DRM would be the plausible deniability of an unsuspecting user "getting hacked" and then their fingerprinted copy ending up as the illegally distributed one without their knowledge or control.
Similarly afaict there are some sketchy closed source programs to remove cinavia,but given we are a decade in, and there is no open source implementation/public description, i would say it lasted better than i would expect.
Definitely not a total success for DRM, but better than many people predicted.
As for Cinavia, if you shift the pitch of the music, you can cause the player to stop recognizing the track. Or, as some people have experimented with, you can compare the audio to a version of the film from alternative sources (i.e. streaming) to detect and patch over potential watermark signals. Or, well, burn a streaming rip to the disc instead of the disc encode.
Including disks with the latest MKB?
> As for Cinavia, if you shift the pitch of the music, you can cause the player to stop recognizing the track
Sure, but that distorts sound, which people dont like.
Like its definitely not totally blocking piracy by any means, but its certainly not totally defeated the way DVDs DRM was. Its still present enough to be a little annoying, which is much better than early predictions thought would happen.
AFAIK, it appears so. Also AACS LA appears to have given up and stayed on v68 for a few years now. Maybe there is a new one at last... maybe. AACS 2.0 it sounds like started from v72 and counting up.
> Sure, but that distorts sound, which people dont like.
True... but if I just burn a stream rip instead to the disc, problem solved. Plus most discs do not have Cinavia.
> Its still present enough to be a little annoying, which is much better than early predictions thought would happen.
From what I can gather... you can just download MakeMKV and rip Blu-rays on any PC Blu-ray drive, no problem, no internet connection required, latest MKB, BD+ completely defeated. It's only annoying if you are trying to actually copy the disc.
It’s a false equivalency.
If anything, I suspect that on net a noisily invasive anti-cheating system would actually increase cheating, as more students would be tempted to see if they could game the cheating system than would be dissuaded from cheating from hearing about a cheat-proof system.
I can tell you with certainty that some cheaters are skittish. I am certain of that because I once wrote a bunch of trig identities on the side of my leg. I was very nervous about it, and only did it because I knew from previous tests in that course that I could get away with it. If the teacher had been walking around during that test, my pant leg would have stayed down.
That is total projection. The consequences of such things cannot even be estimated in such a short span of time. In any case, there is no shortage of studies showing recent massive shifts in quality of education in certain major western societies. Even if you were making the fallacious presumption that what has been will be, you should then presume change over stability.
Can you share 1 study and name 1 major western society you’re referring to?
>For 13-year-olds, the average score was lower in 2020 than in 2012 (260 vs. 263), marking the first time reading scores for this age group declined between assessments.
>Meanwhile, the average mathematics score for 13-year-olds was lower in 2020 than in 2012 (280 vs. 285), marking the first time mathematics scores for this age group declined between assessments.
Not huge, but noticeable.
Any large organization, private or public, is full of checkboxes and CYA. Test proctor software is just another example.
The students being treated like crap isn't just a rounding error here.
I paticuarly like the framing of responsibility.
In modern societies we have pretty fine grained tools for chopping up risk, selling it, trading it, via financial instruments or insurance. In fact when you use these tools sometimes the goal is to vanish the risk all together (which as a result sometimes creates counter-party risk).
Likewise we have a whole cottage industry of chopping up and trading responsibility with very similar mappings to risk. In fact in many ways responsibility is treated more or less like risk itself.
How many problems today though could you follow the thread back to people creating the responsibility equivalent of CDOs.
Maybe we can reframe the issue. Why is it the administration's problem? When I went to college, I went there to learn engineering. I didn't really care about the degree. I was paying to have the prof teach me, not check me for cheating.
Me, I'd make no effort to detect cheaters. If they want to pay $$$$$ to attend university, not bother doing any work, and cheat, I'll take their money.
Make it like guitar lessons. People pay to learn how to play. There's just no percentage in cheating, because you can't play.
The cheaters will see their jobs outsourced, and my salary will be higher.
P.S. In college, lots of students were always doing engineering projects on their own and with their own money. For example, a couple of EEs designed and built a pirate radio station. I wanted to learn how to do that stuff. Cheating is just no fun. And if you don't learn anything, the other students will notice that, and will have nothing but contempt for you.
For example, a doctor doesn’t need to know history very well to be a good doctor. Yet it’s probably part of the undergrad curriculum. So cheating on the test has no consequences if they don’t get caught.
Additionally a lot of jobs are really taught via OJT, and whatever was learned from a degree is mostly unused.
> whatever was learned from a degree is mostly unused.
What a good engineering curriculum does is teach you how to think, which transcends specific skills. It reshapes your brain.
Suppressing cheating is work on behalf of the good faith students. Miserable work it is.
For example, in universities that aggressively go after cheaters, an adversarial relationship develops between the students and the faculty. The students collaborate on cheating, and get social credibility for cheating. Students will brag about successfully cheating to their peers.
On the other hand, at Caltech, we had an Honor System. Exams were not proctored, and students could take the exams whenever and wherever they wanted. The students were on their honor not to cheat. And an interesting thing happened. The students developed a collaborative relationship with the professors. Cheaters were despised and ostracized. If you were cheating, you'd better keep it to yourself - the other students would turn you in, and at that point you might as well leave the university.
"The student-faculty ratio at California Institute of Technology is 3:1, and the school has 66.7% of its classes with fewer than 20 students."
https://www.usnews.com/best-colleges/california-institute-of...
Practical knowledge and rote memorization have held mostly equal status, since out in the field both used to be inaccessible. Our tests are still structured around this, however the reality has moved on and rote memorization is now easily a search away.
I like to see more tests moving to open-book model where test takers are allowed to use all resources, as in work. Of course, this will be harder to administer because schools can't just reuse tests over and over again since they're bound to show up in the internet. Overall, more and more of evaluations will move from testing altogether into some project based evaluation, which I believe is superior in terms of accessing skill.
Let's see how long the old format hangs on.
Otherwise I don't disagree with you that this is where things are heading.
In a few more years we probably won't even be giving traditional grades in school anymore. What would they mean, other than that the student has enough Google prowess to look up the answers?
Simply knowing a generic fact (e.g. "the Civil War began in 1861" or "the atomic mass of Oxygen is 15.999u") is not sufficient. One needs to know how to use that information.
I'm not sure exactly what the distinction is - apart from law being "soft" and requiring argument - but it's deeper than inability to reuse past papers.
Open-book model does not mean you're entitled to all resources, including paying someone to solve all the questions for you, or mooching answers off your peers. Cheating isn't limited to closed book exams, and to that extent, it's still an unsolved problem.
Moving to open book or project model isn't to eliminate cheating (and can't), it's to adapt tests and grading to modern reality.
Bonus if it makes cheating harder.
Therefore the test must change to be more engaging and more in depth, or be replaced with a final project.
I've encountered cases of people looking up questions on StackOverflow for which they themselves wrote the answer many years ago.
My own grades had their ups and downs for various reasons, but when I finally hit my stride, it was with the realization that sufficient memorization would occur as a natural consequence of the work needed to understand the subject matter.
Being able to search for information only gets you so far. Imagine reviewing a design or proposal, or engaging in a technical discussion, if you have no domain knowledge. Have you ever seen a non technical manager try to bluff their way through a technical meeting?
I created my own ring 3 rootkit (user land) and infected my own system to modify registry values to hide remote software to allow complete and total pwnage of the exam itself.
TerminateBlacklist:
mov [esp-4], esi
lea esp, [esp-4]
push offset loc_C81CF9
push ds:TerminateProcess
jmp locret_6CCBD5
EnumerateBlacklist:
mov edx, g_Globals
mov ecx, [edx+0C904h]
mov eax, [edx+0C8FCh]
mov esi, [edx+0C91Ch]
mov eax, [ecx+eax]
mov [esi+edi+28h], eax
add edi, 2Ch
Opening ‘g_Globals’ in memory and browsing through the pointers accessed above such as 0x0C91C brings us to a custom array type containing hundreds of strings.
Taking a look through the struct type and looking back at the array iteration code, wrote a basic structure and dumped all blacklisted processes:
class BlacklistedProcess
{
public:
PCHAR window_name;
PCHAR process_name;
char pad[0x24];
};
Not the most related example, but I was just watching a video about undetectable online cheating in video games with a camera that monitors the screen and a robot that will physically move the mouse to accomodate.
https://www.pcworld.com/article/696507/this-real-aimbot-uses...
Unfortunately for the school, I had experience with Windows programming, and blew the doors off Lockdown Browser in about 10 minutes, with multiple apps and windows open on top of the befuddled browser. And later, I found a way to do it with no programming skill at all.
I did it to make the point that if I could figure it out, who is to say that some rich student's parents wouldn't pay a nice sum of cash for an exploit like that? Or that they haven't? The school, well... just decided to ignore it and keep it anyway because it was a contract.
https://gabrielsieben.tech/2020/10/28/the-lockdown-browser-i...
rant/long comment
First of all, they have a 'different' program for each institution, customized for you to 'sign-in' onto your school's auth page. From an institutional perspective it seems clever, since everything you do will most likely be logged along with your session.
Another annoying thing was that there's different versions of the program for 'each' software. E.g. I used the Canvas platform for some exams and the ALEKS platform for math exams.
And they have the same name under the windows programs so I can only really know which one I'm opening from looking at the desktop.
I believe the Lockdown program is using some lame code to keep the program window 'active'. I never really tried to do any programming behind it. But I do remember that I had some AutoHotkey scripts running for completely unrelated stuff.
On another note, I read their privacy policy and it seems very vague. I wonder if they can get away with letting a stranger view my face for two hours.
And that is not always due to their own wishes: covid lockdowns, unavailability of international flights or other governmental restrictions may be keeping them there.
> Yet even as officials come up with novel ideas, so do the cheats. In February, a medical student at Mahatma Gandhi Memorial College in Indore, a small city, was caught with a skin-coloured Bluetooth device surgically implanted in his ear. A phone linked to the device was sewn into a secret trouser pocket. Last year, ten students taking a trainee-teacher exam were arrested for attempting to use Bluetooth gadgets concealed in the soles of their flip-flops. At least 25 students had bought such footwear from a gang for 600,000 rupees ($7,700) a pair. It is often mandatory for students to remove shoes and socks before exams.
Cheaters get pretty clever.
That would represent a good compromise between an on-site proctor and allowing the student full control over the testing environment.
I wonder if in the future, that will be split. Maybe the place where you study, and the place that tests are certifies your competency in the subject are two different entities.
Medicine, engineering, accounting, law all have some sort of this split where it is actually a different entity (medical board, bar, etc) that certified your competence.
It looks like that's still the case, including in California, where Mr. Mason practices. Of course, word on the street is California's bar exam is the toughest in the nation.
Also interesting: you can create profiles to weigh things more heavily, like weigh a abnormal noises worse than normal if you are suspecting students of using audio calls. They made a profile specific to each of the cheater's method and retroactively applied those profiles to the collected data, and still half of them could not be made to show up near the top of the suspects list. And this is only possible if you already know the exact cheating method your class will use.
The page labeled 288 does make a compelling case for how the spreading of fear, uncertainty, and doubt is effective, leading to less people trying to cheat. It just doesn't stop the cheaters that risk the perceived odds and I doubt this is sustainable once such systems are everywhere and they notice it's safe.
For now, you might as well turn on a dummy system that claims to be real by doesn't actually invade privacy because it's a dummy. Per the results, it would be equally effective and thus achieve the best of both worlds.
In practice I feel like this isn’t even “it failed to detect all six cheaters” but rather “it failed to detect all six cheaters, even though we got them to cheat in ways that are uncommonly detectable and there are much easier ways to cheat that it has absolutely no hope of detecting”.
Study means nothing without breakdown of race / ethnicity (self reported) and age range because in its functional instead of academic gaming for publication environments, the algorithm is designed for a much greater sample size. Of diversity, because the absence of information on this condition the software must account for is woefully naive.
It turned out this was one of the most challenging papers I wrote in undergrad and it forced me to understand my source material on a much more intimate level.
This was 15 years ago and access to information via the internet has only intensified since then. Perhaps our educators need to reconsider from first principles how we assess students in this era.
Further the power of a test already required you to know two parameters: variance and the size of the effect your testing for (difference between means). Proctorio is very careful in not claiming any effectiveness for detecting cheaters, how will you estimate these parameters?
Also iirc, the rule of thumb was for a normal distribution 30 samples is enough for decent strength, 40 samples is enough for general distributions unless you're looking at very small effects, weird distributions or in very noisy experiments.
> For years I've been trying to get any proctoring company to agree to a study where I try to cheat. None have agreed. I've had legal advice not to do such a study without permission from the vendors.
I'm guessing they had to fly under the radar a bit.
0/6 is big enough to go "that's pretty bad".
The second person literally doesn't have to be in the same room, and you could even get fancy with a PiKVM for your assistant to be remote.
The proctor cannot distinguish the two mice or monitors, because the computer cannot either. This is a completely unpatchable hole with no good method of detection.
Otherwise the test taker is in control 99% of the time. The helper doesn't even need to ever click or meaningfully move the mouse.
You could go another level and do a single earbud. Maybe the check for those now that AirPods are super popular, but they sure never needed to see my whole head when we I took remote proctored exams five years ago.
I've had time to think about this, just never bothered to act because the exams I was taking were a joke anyways and I was just looking to get my degree and move on, not make a point on how easy it would be to beat the system.
Frankly, with such low numbers, I would not draw any conclusion at all. Within the margin of error, the human could have detected 0, or maybe 2 cheating students. Who knows. It would change this results dramatically, so you can basically ignore them.
Also, the cherry on the cake is that the human also detected one cheating student who wasn't cheating. So human vs. software, no one wins.
I wasn't familiar with the term. (non-native English speaker, lived in the US for ~10 years)