Hacker News Reader

Comment by anon7725 | Hacker News Reader

> “It’s become clear to me that algorithmic proctoring is a modern surveillance technology that reinforces white supremacy, sexism, ableism, and transphobia. The use of these tools is an invasion of students’ privacy and, often, a civil rights violation."

Must all of the cards in the deck be played at each turn? Things cannot simply be bad/user hostile/privacy invading, etc.

View on HN

It's important to note that none of these things are necessarily done deliberately (though "white supremacy" is perhaps a bad way to express "racism that benefits white people specifically"). Other than transphobia, either the linked article, or the letter linked in that article provide evidence for all of the accusations. Facial recognition software that doesn't handle dark skin well intrinsically treats different ethnicities differently, in this case disadvantaging non-white people. Many of the markers for "suspicious behaviour" that are used to detect cheating are also present in people with both mental and physical health conditions. Dealing poorly with headware has an outsized influence for non-white women (who are likeliest to wear headwear that obstructs the face).

Again, I wouldn't chalk any of this up to deliberate bias against any of these groups, but it's all bias anonetheless.

This is speculation, but many transgendered folks tend to present gender in an non-traditional way, as well as people in the midst of transitioning that may be "in between" presentations in a traditional sense. If you only train on cisgendered faces, you may only be training on sort of "default" gender presentations, facial structure, and the like, which may give a similar disadvantage.

I can totally 100% see how transgender people might trip up some of these things. For the point of this discussion, though, all the other biases had a specific example in the linked article, whereas that one didn't.

Don't you normally expect people to make the strongest case for preventing something? It's pretty common for a lawsuit to bring every claim a lawyer can come up with on the hopes that enough will stick to get the outcome they want.

I would especially consider that in the United States we do not have a broad legal right to privacy but there are potentially much stronger tools available if that software skews negative outcomes towards protected categories like sex, disability, or race. From the perspective of a student, job applicant, etc. being asked to use this, if the legal risks cause an organization to stop using it they'll enjoy that as a win even if the outcome isn't a blanket ban.

>Don't you normally expect people to make the strongest case for preventing something?

Not at the expense of grounded reasoning. When I see poorly substantiated claims, it shouldn't, but it really drags the whole rest of the argument down. The argument presented about lawsuits is actually a great example of why I think that's a broken system. They resolve that issue of lost credibility by considering each issue with total, clear, and mandated separation. Outside of that legal world with very well-defined rules, using such tactics reduces credibility.

I should note, that in this particular case, the claim of racial biases is at least substantiated by a believable anecdote.

EDIT: To clarify why I think the legal methodology is broken, its only because the same principles apply to criminal trials. IMO, prosecutors should NOT be throwing poorly substantiated charges at a defendant just to increase their winning probability and make the required defense more expensive.

> When I see poorly substantiated claims, it shouldn't, but it really drags the whole rest of the argument down.

It should. If someone is willing to make wild, unsubstantiated, claims, it should detract from their credibility.

You're talking about a letter from a U.S. Senator citing published reports in e.g. MIT Technology Review and concerns raised by professional organizations. I think dismissing that as “wild, unsubstantiated” would require at least some discussion of the linked claims.

Why should I reference the particulars of the link when refuting a generalization?

If the parent comment had said something to the effect of "I should give the benefit of the doubt to a sitting US senator" then you'd have a point, but that context wasn't part of their statement.

Edit: Also, frankly, I wouldn't give the benefit of the doubt to a US senator. If anything, it makes the identity politics feel even more irrelevant.

Okay, trying engaging intellectually with the reports rather than just reacting to someone’s one sentence summary of thousands of words. It’s kind of hard to see any definition of “identity politics” which includes the reports but not your emotional reaction to accurate words.

> Okay, trying engaging intellectually with the reports rather than just reacting to someone’s one sentence summary of thousands of words

I wasn't reacting to a summary, I was reacting to an independent premise. The sentence I reacted to, at least the way I read it, was broad to the point of being more of an axiom with which the commenter interpreted the post. As I see it, disagreeing with an axiom is an intellectual engagement. Feel free to counter my disagreement.

> It’s kind of hard to see any definition of “identity politics” which includes the reports but not your emotional reaction to accurate words.

Not entirely sure what you're saying here, but none of what I said was emotional. Simply pointing out that pattern matching is a viable way of filtering other people's thoughts and ideas. If someone makes wild claim, it should change the way you view other claims which may have seemed more rational in their absence. Not really an emotional statement in my mind, but again, feel free to point out which part of this you disagree with and I'd be happy to engage.

As it stands now you aren't really responding to anything I've said, but rather disagreeing that what I'm responding to warrants responding, which is rather tangential.

If you think the accusations are poorly substantiated, then make that case, rather than just complaining that too many accusations were included in the same paragraph.

I don’t have to. Using a Gish gallop of bad arguments doesn’t impose some moral imperative on me to prove every single one wrong.

No, but I think that raises the question of why you think the quoted bit above is a bad argument. It tracks with various reports I've seen linked in HN on this topic over the past couple years.

I know that 95% of the time I see a Gish gallop of identity politics, it's not an argument to engage in at all because even when you do, you're called racist yourself unless you subserviently agree with every aspect of the argument. The identity politics argument is often a tempting one because it allows people to act righteously indignant and feel powerful.

Case in point, here's someone saying, 'How can you even handwave away systemic racism?' in reply to a comment agreeing that bias exists but is not deliberate: https://news.ycombinator.com/item?id=29164295

Also, the identity politics argument seems to hinder a simple, moral argument against surveillance software as a violation of privacy. The logical implication of the IP argument is that this surveillance software would be okay to use if we manage to work out all the kinks.

The difference is that lawyers act in a structured environment with specific rules on how things should be considered - you bring up all the possible claims because if any of them get thrown out, it doesn't impact the others.

It's not the same with general discourse - when you raise a bunch of issues that aren't especially relevant and seem designed to be inflammatory, you damage the credibility of your other arguments. Arguing that test proctoring software is transphobic is such a stretch that it makes you question whether they author has such strong biases against the software that their evaluation of it is just generally too biased to be trustworthy.

> Arguing that test proctoring software is transphobic is such a stretch that it makes you question whether they author has such strong biases against the software that their evaluation of it is just generally too biased to be trustworthy.

It seems like the software is matching people against existing images, based on the issue with the black student, and trans people are I would assume more likely to change their appearance, including as a result of taking hormones and having facial feminization/masculinization surgeries.

The rationally strongest case would be the privacy thing, not the neoliberal-outrage-inducing factors.

> Don't you normally expect people to make the strongest case for preventing something?

Yes, but with some evidence. Otherwise, as a society, this is a bad direction to head in.

https://www.blumenthal.senate.gov/imo/media/doc/2020.12.3%20... references two articles:

https://www.technologyreview.com/2020/08/07/1006132/software...

https://www.nytimes.com/2020/09/29/style/testing-schools-pro...

I have found that two to three strong points far out weight a list of 5-10 weaker points. This extends to the case when the original two points are included in a longer list.

I mean, it's an article about how a technology is negatively impacting marginalized groups, and cites research that backs those claims. It makes sense to cite the groups and practices most impacted by it.

You could read the research and refute it, or you could just bluster about things. I know you tried to expand on it in your comment below, but minimizing the specific concerns raised to "isms" and ignoring that at least two of the references in the articles linked and their references for the actual research addressed at least the socioeconomic portion of it, illustrates that you only applied your surface level perspective and criticism.

Things are bad in particular ways - software that is more likely to falsely punish people of color does support the continued dominance of white people in a real, material way. Noting how things are bad seems useful and important to me.

That being said, there's a rotating list of "badnesses" that are in the zeitgeist and I agree that it's annoying to see them flogged at every opportunity (often w/o much insight).

"software that is more likely to falsely punish people of color does support the continued dominance of white people in a real, material way."

The statement is at best extremely misleading, and at worse, mostly false. It also represents a juvenile, immature, and myopic perspective on reality.

I would suggest reading "Wealth, Poverty, and Politics" or "Discrimination and Disparities" by Thomas Sowell. He has been debunking the "inequality of outcome therefore racism" logical fallacy for decades.

Software that's way more likely to fail you literally because of darker skin sounds way more like "inequality of opportunity" to me.

It’s more likely to fail because…physics. The sensor on your webcam is only so big, and can only capture so much light. Darker faces require more lighting to capture details. Photography isn’t racist, it’s physical limitations that come into play.

A) The major issue is not the sensors, it's the lack of emphasis of ML training data for darker skin.

B) Even if it were simply physics at play, requiring the use of a system known to have physical constraints against darker skin causing failing grades purely on that metric is still pretty racist.

> Photography isn’t racist, it’s physical limitations that come into play.

I wanna pull this apart a bit because I think it's a good opportunity to talk about how systemic bias gets started. Digital sensor evolution is path-dependent. Technologists developed photo-sites that have "enough" dynamic range for most uses before moving on to increase the resolution on a sensor. What exactly is "enough" depends on your test data.

The sensor on a webcam is only "so big" as you say - but how that sensor balances resolution and photo-site count depends on what conditions they consider acceptable. We could build web cams that would see more pigmented faces better - there is no fundamental limitation in the technology itself. It's that a series of decisions have been made over years of development, generally without people thinking specifically about race at all, and we've arrived at a status quo that has adverse outcomes for people with different skin tones.

There was a similar process that happened with film photography[1]. Not that film, as a technology, is unable to capture dark skin - but that the development standards that were tested and distributed were designed for lighter skin.

Like, I agree that the webcams we have aren't intentionally 'racist.' But I do think that the status quo that has led everyone to accept this balance of dynamic range and resolution is reflective of valuing people with lighter skin more.

[1] https://www.nytimes.com/2019/04/25/lens/sarah-lewis-racial-b...

A closely related story that will also resonate with you: early microphones were tuned for male voices, which led to truly a lot of harm to women. It's just a matter of physics, as people are fond of saying. Mediocre technologists make mediocre products that were only validated for people like them.

https://www.newyorker.com/culture/cultural-comment/a-century...

Where did I say that unequal outcomes must, necessarily, be caused by racism? What is misleading about what I said?

It seems like you're reading a specific thing I said about a specific scenario and universalizing it in a way you imagine I might universalize it. I'd love to hear a critique of what I actually said, or we could talk about our views of society in a wider way, but I can't respond to this combination of generally dismissing what I said and attacking what you imagine I might think.

I stand at least partially corrected. I am not familiar with biases in facial recognition software but it looks like a real thing in some cases, caused for instance by lack of diversity in training data sets.

It’s hard to make people care with “bad/user hostile/privacy invading” because those terms have saturated descriptions of behavior that users are okay with. Example: tons of articles mention FB as a privacy invading or user hostile service but it continues to be used by people who don’t really care. Using the same terminology for something that is arguably worse with much higher stakes (algorithmic proctoring that “reinforces white supremacy, sexism, ableism, and transphobia”) is appropriate because it gets the reader to care by illustrating exactly what is possible with algo proctoring.

I sense you’re tired of discussions that mention the “cards in the deck”, likely because you aren’t affected by them and therefore care little for them. That’s honestly fair, but there’s value in writing that way to channel outrage into action.

Is your criticism that these accusations are false, or just that they included too many true accusations in the same paragraph?

My criticism is that including a laundry list of "isms" is a polarizing, low-effort rhetorical device that elides a lot of nuance.

White supremacy: darker skin tones tend to photograph worse than lighter skin tones. Laptop webcams are notoriously crappy and can make this effect worse. Is this white supremacy or physics? Can the test instructions be modified to ask all users to have an appropriate lighting setup (ie lit from above and from the side to ensure that your face is foregrounded properly)

Ableism: for users who require assistive technologies, is it better to take a test in their own space with their own equipment or to travel to a test center and use shared lab equipment? For users with mobility challenges, is it better to be in their own space or travel to a potentially non-accessible testing center?

Sexism: for working mothers, better to take a test in your own home or travel to a testing center and arrange for child care?

Two "isms" seem more relevant but weren’t mentioned: ageism - because fuck boomers, right? Socioeconomics - not everyone has access to a PC that meets the specifications or can acquire one on short notice.

Lots of context is discarded when one engages in polarizing categorical rhetoric. I’m not here shilling for proctoring software but rather for nuanced discourse.

> White supremacy: darker skin tones tend to photograph worse than lighter skin tones. Laptop webcams are notoriously crappy and can make this effect worse. Is this white supremacy or physics?

I’m not a photography expert, but it seems to me that cameras are physically equally capable of overexposing an image and underexposing an image. If a particular camera which is used in a facial monitoring system tends to do one rather than the other, I would ask why that is the case.

Doesn't work like that. Overexposing (with the same light) means having worse SNR due to higher gain.

The choice to use typically terrible cameras in a proctoring system disregarding that it might work ever worse than normally for a subset of people is suspect, yes.

But wouldn't it be just as easy to create a cheap camera that tends to correctly expose dark skin in normal lighting conditions, and cannot dial down exposure enough to prevent light skin from being blown out in normal lighting conditions? If that's possible but cheap cameras tend to not work this way, then why is that not the case?

Yes and no, but mostly no. Lighter skin reflects more light (which means more signal), so it's inherently easier to image (it would be harder in extremely intense light, but getting fast shutter speeds is a lot easier than dealing with not having enough photons). Auto-exposure algorithms tend to work more accurately on lighter skin, too, which could be improved but is generally not something implemented at the hardware level in a webcam as far as I know (software can ask for different iso sensibility and shutter speeds).

The "isms" referenced are less about the fact you can work around these complaints line by line and more about the fact no one bothered to check them before rolling out required surveillance technology for education.

White supremacy is not just bad faith things done by bad people, it is also the assumption that whiteness is the default experience and the failure to account for that not being the case. Similarly with ableism and sexism.

That being said, complaining about the accessibility & inclusiveness of our required surveillance technology does have a dystopian feel to it, lol.

> White supremacy is not just bad faith things done by bad people, it is also the assumption that whiteness is the default experience and the failure to account for that not being the case.

This only works with "new" definitions of racism. It is in fact plainly racist on it's face to demonize a group based on immutable characteristics. It is even worse when actual diversity of though is ignored and people of color are demonized because they don't agree with a race-marxist ideology.

I'm not sure I 100% understand what you are reacting to. I want to understand these two points a bit better:

  It is in fact plainly racist on it's face to demonize a group based on immutable characteristics.

Yes. Where have I, or the study's author, done this?

  It is even worse when actual diversity of though is ignored and people of color are demonized because they don't agree with a race-marxist ideology.

Yes. Where have I, or the study's author, done this?

Would these webcams have shipped if you couldn't see white faces well on them? I don't think they would, and the test proctors would not think of requiring one that didn't. Both the camera makers and the proctors are putting in an assumption that these products are only/primarily for white people, and thus discouraging non-white people is not an issue.

It's not news that facial recognition is best at white male-presenting faces and bad at all the others. ProctorU is also pretty hard to use for the differently abled.

I think you just reflexively dismissed this because you saw a basket of words that normally go along with things you disagree with, but the argument is pretty solid.

> It's not news that facial recognition is best at white male-presenting faces

Really? On the male part? I'd have expected it to do best with women, because I'd have figured facial hair is more difficult to deal with than a wider varieties of hair styles.

When they break even worse when you have black skin, it certainly sounds like a civil rights violation.

It really doesn’t.

> Things cannot simply be bad/user hostile/privacy invading, etc.

In this case, the algorithm is actually bad for all ethnicities [0]. It's just that it's extremely bad for black students (fails half the time) and just regular bad for everyone else (fails a quarter of the time).

[0] https://www.theverge.com/2021/4/8/22374386/proctorio-racial-...

I followed the references to find the actual argument behind calling surveillance proctoring all of the above, and these are the relevant bits[0]:

>At the beginning of a test, these products ask students to verify their identity by matching their appearance with a photo ID. As Os Keyes has demonstrated, facial recognition has a terrible history with gender[x]. This means that a software asking students to verify their identity is compromising for students who identify as trans, non-binary, or express their gender in ways counter to cis/heteronormativity. If a student’s gender expression or name on their ID are different from their current gender expression or name, the algorithm may flag them as suspicious. When this happens, they may have to undergo another level of scrutiny to authenticate their identity, an already common and traumatic experience for trans and gender non-conforming students. If these students are not alerted of this possibility before the test begins, it may force them to either discontinue the test and risk their grade, or out themselves to their course owner when they may not want to, risking more trauma and discrimination including being denied financial aid, being forced to leave their institution, or have their lives put in physical danger.

>The Eugenic Gaze is a combination of white supremacy, sexism, ableism, cis/heteronormativity, and xenophobia. When we apply the Eugenic Gaze using technology, the way we do with algorithmic test proctoring, we’re able to codify and reinforce all of those oppressive systems while avoiding equity-based critiques because of our belief in the neutrality of data and technology.

Their recommendations are quite reasonable:

>Don’t use algorithmic test proctoring. Instead, focus on pedagogical techniques that you can use to design assessments, online or in person, that draw from personal experience or require students to apply concepts in unique contexts. If you have to use algorithmic test proctoring, make sure students know about the test settings and ID requirement well before they take a test, and assure them that you will not take any behavior flagged as “suspicious” into consideration that isn’t described explicitly in the syllabus.

The GP link[1] instead calls out "Facial Recognition Tech", and "Algorithmic Proctoring" as being too biased and follows up with a petition[2] to ban these entirely.

[0]: https://hybridpedagogy.org/our-bodies-encoded-algorithmic-te...

[1]: https://library.auraria.edu/news/2021/why-online-test-procto...

[2]: https://www.sheaswauger.com/post/petition-to-ban-facial-reco...

[x]: https://ironholds.org/resources/papers/agr_paper.pdf "The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition"

Which one of these is inaccurate?

The claim of "transphobia" is extremely weak, for one. Clicking through the maze of links, the argument appears to be that because algorithms that attempt to guess gender based on photos sometimes guess wrong when trans people are involved, all attempts by computers to look at faces must therefore be transphobic, even if those computers do not attempt to guess the gender of the person whose face they're looking at.

Misgendering people intentionally is transphobic, and systems of this nature behave this way. For many trans people getting an ID to correctly reflect their name and gender is extremely difficult and can go for months/years without it being corrected; companies like Proctor U don't care about the problems this causes: https://www.teenvogue.com/story/exam-surveillance-tools-remo...

https://library.auraria.edu/sites/default/files/images/fancy...

The quoted person expands on their arguments on those topics in the linked article. If you think they're needlessly "playing a card" why not engage with the actual argument?

Yes because to do otherwise would be exclusionist.

Reply on news.ycombinator.com