Microsoft patents tech to score meetings with body language, facial expressions | Hacker News Reader

Microsoft patents tech to score meetings with body language, facial expressions

geekwire.com

51 points_gtly46 comments

I like how thinly veiled the doublespeak is these days: “Let me be clear: Productivity Score is not a work monitoring tool,” he added. “Productivity Score ... focuses on actionable insights about the ways in which people and teams are using the tools so you can make improvements or provide training to further your digital transformation.”

Another way to describe it would be a clear lie, by their own definition. What other way you could you know how people are using tools then by...monitoring the use of said tools?

The PR bullshit wording itself should tell.

My fear is that these kinds of tools won’t be used to gain more data and make better decisions, but that these tools will be used to gather all kinds of random data to justify firing decisions.

Gather enough types of data, you’ll look bad by one metric, and companies can justify firing anybody by pretending to cite the data.

And never mind the fact that MS is a vendor, and this score is implicitly stating that using their products = productivity. If you believe that, you also believe that Microsoft’s collaboration tools are perfect, and the only reason an employee is not highly engaged with the O365 suite is that the employee is not productive.

Was thinking about that the other day. The purpose of many of these algorithms is to justify deliberate human decisions, not to provide new insight. Similar to business consulting firms that are brought in to recommend "restructuring" when the company wants to fire a bunch of people.

They're saying it; not us!

It’s just another tool in the toolbox. For instance, sometimes high priced consultants aren’t brought in to gather info and make the best recommendations. Sometimes they’re brought in to give the boss some kind of backup and deflection if things go wrong. But often they would recommend exactly what the boss wants: they know where their bread is buttered.

I think it will just lead to meaningless metrics where management advises everyone to constantly improve body posture so the metrics look better.

As technology people and capitalists, we seek to optimize everything for performance and profit.

Right up to squeezing the humanity out of everything.

I think This is how IBM "deal" with their older employees (designing scores to make them easier to fire, that is)

The first thing that comes to mind is Hirevue[1] rating job candidates based on metrics like eye contact. Generally I see this product as automating the process of discriminating against people with autism and see it as deeply immoral.

[1] https://www.washingtonpost.com/technology/2019/10/22/ai-hiri...

That's just signaling; it's a way for enterprises to tell graduates that there's an oversupply for the position they are applying to.

It will be interesting to see in what new ways this disproportionately negatively affects those on the autistic spectrum, or with other similar behavioural traits.

Not just behavioural traits, just wait until it turns out their AI model classifies every facial expression from their black employee as "confrotational".

Obviously, the "solution" is to use AI to deep-fake your own face, but with the negative emotions stripped out! When in doubt, add more AI/ML /s

Using a firewall personality filtered through your smart camera is already a thing. You can use the snap camera to apply a lens to reduce the emotional tells.

Before the Snap camera, folks were taking pictures of themselves looking attentive and replacing their meeting image so you couldn't tell they weren't moving.

I helped build something kind of like this in college for a research team that was aimed at facilitating in-person arguments between 2+ people. It took speaking tone, body language, and facial expressions as input to judge how productive the conversation was in terms of resolving an argument, and would offer spoken interjections to steer the conversation toward more productive / less aggressive paths.

It was significantly more effective when participants went in blind and didn't know how it "knew" how they were feeling, but usually after 30-45 minutes of observation people would start to pick up on how it worked and often tried to manipulate it instead of actually trying to resolve the conflict.

Our application seemed good in theory, but worked horribly in practice. Hopefully Microsoft can figure something out that is net-beneficial in some scenarios without going full-dystopian.

> Hopefully Microsoft can figure something out that is net-beneficial in some scenarios without going full-dystopian.

The chance of Microsoft not going full-dystopian is practically zero. It's an internal-political + trust issue, and they've demonstrated copious times they have no regard for human ethics.

I have toyed with prototypes for using sentitment analysis as a submission filter in an online discussion system.

I think rather than being powerful, the ML should just be a facilitating agent, like giving people a turn to speak or holding votes, etc. Basically a skilled impartial secretary to apply some of the techniques of Robert's Rules of Order (or something similar).

I just finished watching the second season episode of The Good Place "The Trolley Problem" with my daughter, spoilers ahead if you haven't watched to the end of season 1, so don't go any further if that kind of thing is important to you:

in the show, Michael, a demon, tells Eleanor, a human, that he is a superior being that can see in 9 dimensions and because of his superior senses and understanding of the universe knows that she is about to fart and lie to try to cover it up.

so what I'm saying, I guess, is that Microsoft wants to build the Bad Place, and then try to convince us it's the actual Good Place.

This is so stupid. Have these people not heard of Goodhart's law?

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Not only is the application of these surveillance technologies morally reprehensible, it's also a complete joke from a performance standpoint. The only thing it will do is turn meetings into beauty contests. If you've ever asked yourself why congressional hearings of famous CEOs are such a clowncar, pay attention to the number of cameras in the room.

A patent on this invention pretty much removes any incentive to develop it further, at least in the foreseeable future.

>This is so stupid. Have these people not heard of Goodhart's law?

Yes, but they have also heard another law: "Any measured metric, no matter how BS, can be useful in justifying what you want to do and give you control over others".

"You can't manage what you don't measure."

"How else are people going to make decisions? Their personal biases?"

An objective KPI for sociability will be manna from heaven to these people; it was the last thing they were stuck with human intuition on.

I agree that this will be radar for sociopaths, so it should be a hit with leadership.

On a plus side, this should stop folks from making 3rd party products that rip video feeds out of the meeting and run realtime emotion analysis.

The one positive version of this I would like is if you could score anonymously. Then it might be nice to have an independent tool tell management certain meetings have been a waste of everyones time and nobody has been paying attention

It's easy to see the potential for abuse for any new technology, but it's much harder to come up with actual uses and worthwhile applications.

I don't know if this technology will do any good, but I appreciate this comment for finding a good use (hopefully one of many) for it.

It is just a matter of time before execs at megacorps use these arbitrary metrics from Microsoft to evaluate knowledge workers. Of course, there will be no differentiation between quality and quantity of employee interactions. And, of course, the execs themselves will not be subject to these metrics.

A meeting room filled with endless cameras and censors to record and judge every facial expression and body movement I make...

Do they purposefully make these patent diagrams look as dystopic as possible?

There seems to be a lot of employee surveillance technologies that are being developed and pushed by Microsoft. I recall them advertising what they have in mind for AI in the whole Office ecosystem and they were both depressing to think about from an employee perspective and also unsound from a technical perspective. I think employees should demand more transparency at least and know how their everyday behavior is being monitored and analyzed, especially as it relates to emails and messaging.

I experienced some of their meeting monitoring stuff several years ago on a visit to Microsoft Labs. I noticed the meeting score stuff and made a joke about how the meeting room was watching us. One of the MS people responded with "Yes, it is." And then they showed us the "telemetry" the room was taking.. IDing us with facial recognition, guessing at our moods, etc. I definitely felt uncomfortable for the rest of the meeting.

It’s an arms race, hopefully a group of smart folks will open source a OBS (or similar) plugin to manipulate or completely mask minor facial expressions.

From the owners of GitHub and LinkedIn.

I’ve been wondering how long it will take them to “cross the streams,” then sell the result to employers.

So we have a Voight-Kampff test to find employees who don't give a shit (https://news.ycombinator.com/item?id=25229291)

I think there's a valid conversation about how this is just a straight up dystopian bad idea. However, I think one thing also worth mentioning is that essentially everything we do on computers looks like this: We take something we know how to do, and we make the computer do it faster. We don't know how to score meetings using body language and facial expressions - or atleast I would be incredibly sceptical of anyone who claimed they could do so without incredibly intimate knowledge of the people involved. Any manager will tell you there are people who are productive when talking, people who are productive when quiet, and people who are unproductive when talking or unproductive when quiet.

What seems likely is this system is going to invite extroverts to your meetings. My gregarious bizdev lead is going to be coming to our standups to boost our scores.

Let's just lay aside the concerns about this being a dystopian nightmare, it's also going to be crap.

If my company announced that they were going to use this, that announcement would be followed in fairly short order by a brain dump of all the stuff I've been working on for them and then my resignation effective immediately.

I am reading this after waking up... kinda wishing I did not wake up. What on Earth is going on? We, the people on whom they do this crap cannot (should not) tolerate this. Hopefully someone has ideas about what to do against this kind of shit other than not work for Microsoft or wherever this technology is used, like what is happening to those if this becomes widespread? We cannot all just expect others to not work. We need other solutions, and we kind of need them fast because it is getting out of hand. We frown at China, but this sort of thing is not any different than for what we frown at China, is it?

"Policy"-style debate[1][2]. What happens when metrics must be gamed to win.

State as many points as you can while gasping for breath. Coming to coworkers near you.

[1] https://www.youtube.com/watch?v=0FPsEwWT6K0

[2] https://www.youtube.com/watch?v=murvOaHB66A

>The system uses cameras, sensors, and software tools to determine, for example, “how much a participant contributes to a meeting vs performing other tasks (e.g., texting, checking email, browsing the Internet).”

It implies your company is recording what's going on your computer and on your phone, too, not just from body language and facial expressions!

So toxic/dystopian.

Glad I work for a company that does not require the use of a Webcam, and I never ever turn my on for meetings

Just because they don’t require it doesn’t mean it isn’t being considered a negative or positive thing against you in the future.

Well considering that in my dept fewer than 5% of people also use their webcam for meetings I think I am in the majority not the minority

Don't work at any company that utilizes tech like this. Brain drain.

So if the meeting score was low, does that mean that no useful decisions were made?

How is this better than just having a survey after every meeting?

I'm guessing that this didn't come out of Microsoft Research?

As long as I can practice on the system to know what facial expressions to produce.

Reply on news.ycombinator.com