Facial recognition can predict person’s political orientation with 72% accuracy

There's an avalanche of people commenting on this who didn't bother to check the article before raising their methodological objections, so let's get these out of the way here.

- Yes, they controlled for objects appearing in the pictures that might indicate political affiliation. The images are tightly cropped around the face. See Methods.

- Yes, this is significantly better than both a coin flip and a human classifier. They gave the same test to humans, who did much worse than the model. See Abstract, Introduction, and Results.

- Yes, this is doing more than just detecting a person's race, age, and/or gender. The classifier is still accurate when they compare people with the same race, age, and gender. See Results.

If you want to discuss actual limitations in the study, here are some the author points out:

- "A more detailed picture could be obtained by exploring the links between political orientation and facial features extracted from images taken in a standardized setting while controlling for facial hair, grooming, facial expression, and head orientation."

- "Another factor affecting classification accuracy is the quality of the political orientation estimates. While the dichotomous representation used here (i.e., conservative vs. liberal) is widely used in the literature, it offers only a crude estimate of the complex interpersonal differences in ideology. Moreover, self-reported political labels suffer from the reference group effect: respondents’ tendency to assess their traits in the context of the salient comparison group."

This isn't a study that shows that people's faces indicate their political leanings.

It's a study which shows that pictures that people select to represent themselves publicly have features that indicate political leaning.

I think the key line is how much better this system is than humans attempting the same task:

> Political orientation was correctly classified in 72% of liberal–conservative face pairs, remarkably better than chance (50%), human accuracy (55%), or one afforded by a 100-item personality questionnaire (66%).

This isn't a matter of "recognize that old white people are conservative", because people will do that already, and they know all those biases. And the system doesn't lose much accuracy when comparing otherwise similar people.

This system is picking up on things we don't notice. Maybe it's the photos themselves (they are self-selected), maybe it's micro expressions in the face, maybe it's something else entirely.

But damn it's neat and maybe frightening. Imagine if your next hiring manager had a quiet little camera in the corner and chose you based on your predicted approval of unions.

"The dating website sample was provided by a popular dating website in 2017. It contains profile images uploaded by 977,777 users; their location (country); and self-reported political orientation, gender, and age."

It doesn't sound like people were consenting/aware that they'll endup on a facial recognition study.

But I'm not even surprised dating website would sell their such data

> Their facial images were obtained from their profiles on Facebook or a popular dating website

So, this seems to be doing profile picture recognition, not facial recognition. It's not like they're saying there are physical features in your face that give away your political affiliation - that is to say, putting two people's bodies under the same photographic conditions would probably not create this kind of signal.

What this is probably training on is the cues for cultural values that we self-select in our most deliberately promoted images of ourselves. If you have a carefully-chosen profile picture, it probably includes signals of what's important to you, especially if you're using it to attract people with similar values.

At some point in grade school I was home sick for a few days and ended up watching C-SPAN for several hours on end. I don't recall exactly what was going on politically during that time, but many congresspersons were standing up and giving speeches for a few minutes at a time.

I eventually started a game in my mind where I'd try to guess their political affiliation before the chyron appeared. I'm pretty sure by the end I was getting it correct more than 50% of the time. Everyone was dressed similarly, but I remember looking closely at their tie patterns and hair cuts as clues.

I only skimmed the paper, so I'm not claiming to know much about it, but one thing to keep in mind here is that a fair coin has a 50% accuracy using the same terminology as the headline. I'm not saying 72% is not an interesting achievement, its just that "you can do about 50% better than random chance" describes my gut feeling about how much you could actually see in someones face.

> the relative universality of the conservative–liberal spectrum

Grey tribe members may beg to differ

https://www.gsb.stanford.edu/insights/rise-liberaltarian

Can we run the model in reverse (eg. deepdream) to see what the stereotypical liberal/conservative looks like?

I want a facial recognition tool that tells me what information I am leaking. We need a haveibeenpwned for facial recognition.

I question the dataset used for this and the basis. They used "self-reported political orientation, age, and gender" and "facial images (one per person) were obtained from their profiles on Facebook or a popular dating website". I first question the ethics of what sounds like a Facebook scrape. Second, I wonder how well they normalized across the terrible filters and frames and variations in pose. Finally, I question the basis in regards to the ability to gauge one's "openness to experience" let alone, say, "opinion on immigration policy" from micro-expressions in the face (or at least those which could be interpreted by a VGG based facial recognition algorithm).

Edit: One last rant about this palm-reading-esque pseudoscience: I hate that this was put out in the universe, and thus potentially giving the wrong person ideas.

I assume there's no way to actually verify how someone may choose to vote, assuming there's no record of that?

I think there's huge value now that everything is being sent into a "machine" or "the algorithm" in fucking with it.

Order sex toys from Amazon, show them you're into outrageous books and fool them into creating a fake profile of "you", based on your spending, browsing and other data you generate.

I'd love to ask a machine what it knows about me, how accurate it is, and then switch it all up. I'm too old to vote now (and will probably be dead soon) but I'd love to pick a position completely unexpected just to throw it off.

Poison the well

Evil plan: release this model as an app, heavily branded to skew usage towards my political enemies, and suggest it to be used to find "<opposite side> infiltrators in your midst".

Accuracy issues are a feature not a bug! Now you've sown a bunch of discontent and suspicion within their communities.

I find this fascinating, with no intention to start a political battle, how might somebody’s biology influence their political leanings?

I wonder if the study accurately reflects the lack of political common ground in the population. Are only 28% of us “moderates”?

It’d be interesting to do the same analysis with photos of people taken 10, 20, 30, 40,... 100 years ago to see if the intersection (28%) grows/shrinks historically. Were we always so easily politically separable by our appearance alone?

Has our appearance always revealed our political leanings, or do we now dress in order to make a political statement?

I wonder how much data about a person can you squeeze out of a high-definition video, analyzing not only face, but grimaces, tone of voice, rhythm of speech etc.

That should give you a lot more hints than a still image.

The worst possible uses: filtering out undesirable people applying for jobs or for college, firing people who are suspect of belonging to a different political tribe.

It seems strange to me that the authors of paper seem to have made no attempt to determine which factors the model is using to differentiate between images. Or even to ask the human comparisons how they obtained their greater than chance results. Instead, both the humans and the computer model seem to be considered to be considered perfect "black boxes", which give entirely inscrutable answers.

It's not that it's always trivial to figure out what's happening inside a model like this, but I'm surprised that no attempt seems to have been made: modifying images and seeing which increases the scores, bringing in a co-author who actually understands what the model is doing, seeing if they can obtain similar results with a more interpretable model. Am I wrong, and in fact fining out what's actually happening is impossible, or of no benefit?

May be the world polarising into two kind of extreme polarisization: one with a sense of humor and one without it. And it has a noticeable effect on the face. I'd love to see few samples of faces from each class.

When are people going to stop rebuilding high tech phrenology?

I just wish people called it 'correlates' instead of prediction since ML algo's often fall in the correlation category, not the predicting one.

So, unless I read the abstract wrong, this is identifying age, race, and gender, then making categorical qualifiers based on those distinctions. While using ML to do the facial recognition to distinguish a person's age, race and gender is neat - categorizing their political affiliation with a 72% accuracy rate is fairly nominal, given the tools used by modern parties to garner donations and directed online advertising - no offense.

So now I want to know 1. what is the eigenface for each of the two clusters? What does the perfect liberal look like? 2. What is the political orientation of https://thispersondoesnotexist.com/

> These self-selected, naturalistic images combine many potential cues to political orientation, ranging from facial expression and self-presentation to facial morphology.

I was going to ask how they controlled for the presence of compound bows & deer, trucks, and tank tops in the photos, but this implies they did not. Another one would be lighting palette, since that's going to be biased to regions as well. There is something to be said for it, as I'd say I have a %72 chance of guessing someones political orientation by looking at them as well, which someone once explained to me as being the effect of testosterone levels on the region around their eyes, but that sounded like folksy bro science.

How would we collectively react if ML proves accurate in classifying anti-social traits or worse ? Mythomania, narcissic perversion, pedofilia, etc ?

Employers, condos, schools, many communities will want to ML-screen candidates, be it officially or not..

It's an easy trick when there's only two options: red and blue.

- The rest of the word.

I haven't read the paper yet, but it's not that surprising. I think it's about sociology more than biology.

In the US, African Americans vote overwhelmingly for democrats for example. Skin color therefore becomes a very good predictor of political orientation. You can probably extend this to states being populated from various migration waves, say 'people who look like Danes vote for republicans because state X was populated by Danes and votes Republican' . Carry this across generations/education, and you may have an explanation.

Assuming this analysis has no problems with method, it's likely that the separation of any two groups on a continuous spectrum cannot ever reach 100% accuracy. In fact for such multivariate groupings as opposing political persuasion, 72% may in fact be as good a score as is possible.

Representing the conservative and liberal groups as gaussian mixes of multiple atributes, I would expect those two peaks to overlap. Perhaps the real surprise is that they overlap no more than 28%.

That headline is a bit misleading. The facial recognition has to see two faces known to have different political orientations in order to deduce which face is which. The 72% classification correctness is about what one would expect if only 25% of persons had visible features that identified their political orientation and the other 75% were completely inscrutable.

The problem wit this is bad interpretation, people look at it and think oh this means political orientation is somehow inherent to biology or some such nonsense. Any data that can make predictions is just a sign that in this data there are some patterns, but there is never a straightforward interpretation for it.

Also, whoever makes the first app for this will go viral 100%.

So... now, is it because people that came from the same regions a long time ago all happened to tend towards a certain political leaning and that just has stayed within families for that long?! Like a culture thing, but within families?

In general, though, its amazing how little control we have over who we are despite the feeling that we are in control of it.

Isn't it possible in many ML models to then synthesize of the images of the features that most strongly predict the orientation? Like an archetypical "conservative" or "liberal". Could this help identify if the model is picking up on something like facial expression, or facial hair?

> "Accuracy remained high (69%) even when controlling for age, gender, and ethnicity."

So if I just assign the majority label to all of the population of a given demographics group, I would get the same result right? i.e., predicting "left" for all minorities under 30. You would also get ~70% accuracy.

Reality please don't bite me. This will probably be an Oxymoron :)

An interesting question, to me, is: how would you (personally) react if such an controversial and stigmatized factoid would be proven true at more than 5 sigma?

Would you be shocked? Or would you accept the outcome? And which of both would tell what of yourself?

The predicted 'answer' for the vast majority of people is going to be, basically, left or right (politically speaking), for maybe 80 or even 90% and more people, a 50/50 choice. How far does the 72% drop for correctly predicting the smaller political factions, I wonder?

Haven't seen this mentioned yet, but ResNet-50 is an old model. I would probably expect multiple-percentage-point gains from using a better (or honestly just larger) architecture and better training methodology.

Throw an ML engineer at this task and you could probably do * way * better than 72%.

This is a fascinating result. On the other hand it is also telling .. and maybe dangerous? If we can predict political orientation, can we predict if someone is gay? If he is jewish? What comes next?

Would't this legitimate all these voices who ever said I can tell you by just looking?

And I always thought it was funny how in classic books from authors like Jane Austen they’d always talk about a person’s physiognomy, and how you could draw strong conclusions about them from it. Maybe it wasn’t so bogus after all?

Is the model available? Would be interesting to see how it classifies members of Congress, governors, state legislators, etc. Assuming politicians weren’t used for the training set? (Also assuming Kanye wasn’t either?)

I can see making inferences from dress, makeup, style, etc. But, this is surprising; “facial morphology”

This skims somewhat close to people who tried to infer criminality from facial and cranial characteristics... a century ago

theory: seeing as these are pictures that were very specifically "taken" in a given context - with a camera, by someone, for something (dating, facebook, etc.), would it be possible that the algorithm is picking up differences more in how people relate to those conditions, vs facial morphology? For example in the example photo, i would have also guessed that the model was a liberal. But I think it was more about the something in the WAY she was smiling and posing for the camera, and i guess also her specific grooming - hair and makeup.

Is it also 72% accurate for income level?

I think the incomes of people on the right skew lower, or they have fashions for makeup, facial hair etc. I'd be interested in an adversarial attack on the classifier.

> Had the political orientation estimates been more precise (i.e., had less error), the accuracy of the face-based algorithm could have been higher.

That is an extremely loose assumption.

I am curious how much additional information, if any, is learned from a person's face. From a full-body photo, with the face blacked out, what can be learned?

They claim facial morphology was one of the predictors. Pretty wild if true, but how does genetics / heredity play into political leanings?

> Their facial images (one per person) were obtained from their profiles on Facebook or a popular dating website.

Not sure I’m entirely comfortable with that.

The article discusses the underlying correlations at work but I think the title is a bit sensationalistic. Would expect more from Nature.

My political orientiations have changed a few times over the last 10 years but my face hasn’t. This can’t actually work...

> Accuracy was similar across countries (the U.S., Canada, and the UK)

This sounds more like people who are from minority groups (or who look like they are, to a computer vision algorithm) are more likely to agree with left-leaning policies, which probably has more to do with the policies in those countries than it does with any sort of genetic features. I feel like this might not work as well in non-Western countries, for example.

Altough random guessing is 50% right?

Is this surprising?

We all know there is a demographics/ethnics factor in people's political affiliation.

Has there been any attempt to do the same kind of work for IQ prediction ?

This is hilarious study. This is like Zuckerberg calling us dumbfucks.

Sigh - modern day phrenology, but wrapped in AI. Just what we need :p

What's a useful application of this, other than more advertising?

Surprising, yet not surprising. As the article mentions:

> Both in real life and in our sample, the classification of political orientation is to some extent enabled by demographic traits clearly displayed on participants’ faces. For example, as evidenced in literature and Table 1, in the U.S., white people, older people, and males are more likely to be conservatives.

Most people can predict a person's political orientation of their own country with >50% accuracy as well just by looking at a face. Black or latino? probably liberal. Old white person? probably conservative. If you can see more than just their face it's even easier (wearing religious paraphernalia? LGBT paraphernalia? etc?)

What I thought was interesting was:

> The algorithm could successfully predict political orientation across countries

I was under the impression that "liberal" and "conservative" had different meanings in UK vs. USA so how could it do this?

> The highest predictive power was afforded by head orientation (58%), followed by emotional expression (57%). Liberals tended to face the camera more directly, were more likely to express surprise, and less likely to express disgust. Facial hair and eyewear predicted political orientation with minimal accuracy (51–52%).

This is really interesting. I never considered head orientation or expression to be a factor. Then again, it sorta makes sense. Speaking very generally, liberal leaning people on social media probably tend to be more likely to post pictures of themselves in a humorous or "soy face" expression, and conservative-leaning types may try to look strong or aggressive.

Also, I automatically assumed features like "handlebar moustache" would mean more likely to be conservative, but it sounds like facial hair wasn't as big a factor.

A data point suggesting bias is more accurate than a coin flip...

Too much close to a Phrenology 2.0 IMAO

If you can guesstimate someones age bracket, gender, and whether or not they're white you can also probably hit a similar accuracy.

Who knew. Being a progressive/moderate makes people more attractive...and improves your sense of humor.

So basically not at all.

could we please not

As someone who's studied facial expression in cartooning and tried to track down how that's done through face muscles, this tracks.

Not only that, it is hackable, and somewhat mutable… and I think that can reflect back into one's general attitude on life. I'm going to share some personal notes on my own face hacking done to serve my purposes as a youtuber and open source coder…

The key phrase here is, "The highest predictive power was afforded by head orientation (58%), followed by emotional expression (57%). Liberals tended to face the camera more directly, were more likely to express surprise, and less likely to express disgust." People will respond to you based on what your face is doing, and be more or less favorably disposed to you if you 'match' where they're at.

Guy Kawasaki's on record as trying to maximize his ability to Duchenne smile (crinkle the outside edges of the eyes as your cheeks go up) in order to better influence others. You can make special efforts to dry your skin there, the better to form heavy wrinkles that can come into play, signalling affability and well-disposedness as a proper Duchenne smile would do.

But there's another area. If you fret a lot, or glower, your brow comes down and wrinkles form where your brow meets your nose. This signals suspicion, disgust, hostility. My face-hacking involves putting Nivea cream there and on my forehead, keeping that skin more flexible and mobile, for a more open affable look. But if you're targeting a conservative audience you can do the opposite: look at Tucker Carlson sometime. You can cultivate a world-weary scowl and it will increase your trust with people sharing a similar facial expression, and tend to concentrate your viewer's expressions into ones similar to your own (while you tell them scowl-worthy things), so long as you have their basic trust to start with.

This is all very malleable. Very hackable. You can do it on purpose. I don't know if Tucker Carlson does scowl exercises, but I know if he botoxed his brow scrunch, he would be less effective as a political commentator, because he would be telegraphing the intended reaction to his information more weakly.

We're looking at a general connection between human resting facial expression, and human overall outlook on life. I didn't expect to run across this study but I find it absolutely plausible. Almost axiomatic. You can even frame it in ways that appear to favor one political side or the other, but the underlying principle tells us a lot about how political orientations arise.

"only"

This study proves that the liberal surprise face meme is real.

Who wants to take bets on how long before the paper gets cancelled and forced retraction?

It is like saying - My intuition usually works great. 78% of the times I have been successful with my assumptions/speculations.

Generally speaking, these sorts of things don't make any sense to me. People change as they age, as they get new experiences, when they live in different places. How can someone's face tell you what their political orientation is?

When I lived in Texas, I was considered a liberal. When I moved to Chicago, I was considered a conservative. When I moved to Seattle, I was considered a conservative. When I moved to the desert southwest, I was considered a liberal. Nothing about my face changed. Just my address.

Oh cool, are we going to bring back phrenology as long as its an AI powered black box. Sounds wonderful. It's going to be impossible for any of this to be abused. I say we should give AI power to a select few people who can make sweeping decisions in everyone's life. Oo Oo can we bring back eugenics too? What's wrong with an AI powered method of selecting the best genes to continue the human race. Then we can sterilize the degenerates the almighty algorithm picks out.

I swear to fuck. All this praise to repeat the sins of the 20th century all over again? But oh no, it's okay because it's done by "software engineers".

That and you "atheists techies" are just as fanatically religious as jihadists. Instead of a deity, you worship silicon valley and algorithms.

Of course tech companies can be trusted with our data.

Theres nothing to fear from putting your life on social media if you have nothing to hide.

A select few should have absolute say on what we are allowed to even consider "free speech".

Tech companies aren't in it for profit, they're in it to save the world because they're the "educated elite".

Just... why is this not being shot down? Are you that blind to where it's going to lead? Its literally the same steps every other totialirian psychopath took. Mass identification on bullshit pseudoscience. Then comes the extermination.