Eye Contact Correction: Redirecting the eyes to look at the camera

I feel hardware technology can improve further to allow under-the-LED-display cameras .... so that we can actually look at both the camera and the screen at the same time.

(There are fingerprint sensors under mobile screens now ...and I think even some front facing cameras are being built in without sacrificing a punch hole / pixels. There is scope to make this better and seamless so we can have multiple cameras if we want behind a typical laptop screen or desktop monitor.)

This would make for a genuine look-at-the-camera video whether we are looking at other attendees in a meeting or reading off our slide notes (teleprompter style).

There would be no need to fake it.

More philosophically --

I don't quite like the normalization of AI tampering with actual videos and photos casually -- on mobile phone cameras or elsewhere. Cameras are supposed to capture reality by default. I know there is already heavy noise reduction, color correction, auto exposure etc ... but no need to use that to justify more tampering with individual facial features and expressions.

Videos are and will be used for recording humans as they are. The capturing of their genuine features and expressions should be valued more. Video should help people bond as people with as genuine body lanuage as possible. Videos will be used as memories of people bygone. Videos will be used as forensic or crime scene evidence.

Let us protect the current state of video capture. All AI enhancements should be marketed separately under a different name, not silently added into existing cameras.

I have been working for a company which allowed full remote work without any qualms since before COVID and nobody did video calls back then. Since we end up on site in secure environments we also just get told to disable the camera in the BIOS as part of our laptop hardening.

For things like bi-annual meetings with your manager you would go into your local office.

It's from January 2023, so I don't know if they've improved it further since then.

The video conferencing software providers have been way to slow to put whoever is speaking top-center (near where the camera typically is).

The normal thing is not to uninterruptedly look at a person (which the camera is supposed to be). For example when you make a gesture of trying to remember something by looking somewhere else.

And if they do, do you like it?

I'm autistic. I do not make eye contact easily. That's part of who I am, and a useful signal for the people I interact with that I might be neurodiverse.

Something like this could mask or hide that autistic trait, and make me appear allistic. And, for some autistic folks, maybe that's desirable. However, I find the traits that I have around communication to be a deep part of me. Not making eye contact is nothing to be ashamed of, because being autistic is nothing to be ashamed. The introduction of tools like this could lead to pressure to use them to conform to some standard of "normalcy" that people expect.

While the technical achievement here is neat (and other commenters pointed out places where it's struggling to look good), but there's a sort of meta impact that it can have on people who fall outside of normal ranges.

Building further on this idea, I wonder if instead of changing the image to look at the camera, we could change the "camera" to be where we're looking.

In other words we could simulate a virtual camera somewhere in the screen, perhaps over the eyes of the person talking.

We could simulate a virtual camera by using the image of the real camera (or cameras), constructing a 3D image of ourselves and re-rendering it from the virtual camera location.

I think this would be really cool. It would be like there was a camera in the centre of our screen. We could stop worrying about looking at the camera and look at the person talking.

Of course this is all very tricky, but does feel possible right now. I think the Apple Vision Pro might do something similar already?

I sometimes use an Elgato Prompter to better enable eye contact during meetings. The camera and lens is mounted behind the screen so looking at the screen is also looking at the participants. The downside is that the screen is tiny and you leaning forward to read, say, a document does not look that great on camera. So either you have to zoom it substantially or read it on another screen, thus looking away from the participants. In this case though, you are not looking at the participants and faking that eye contact in this case would be kind of weird.

One thing I've always wondered is if this could be made to work for group video chats - depending on the tile you are looking at, that person would know, so you could tell who is paying attention to you, or even exchange a furtive glance with a colleague in reaction to someone else said like IRL. Even harder, but also cool would be updating the gaze dynamically so you could tell what they were looking at in your scene - say you have a whiteboard behind you and you can tell when the person is making eye contact with you vs looking at something you drew on the board.

Original dev make it so! :)

Just a heads up – your main website is showing an error. You might want to fix it since your post is gaining traction. Here's the link: https://www.sievedata.com/

The error message reads: 'Application error: a client-side exception has occurred (check the browser console for more details).'

I think it's great that this is labelled as "correction" as in a means of optional postprocessing when it's convenient. Nvidia implying that it's something we should enable by default rubs me the wrong way, but then again, I don't spend my day stuck in virtual meetings.

But the resultant video has a tad bit of uncanny valley going on.

I'd rather learn from the guy on the right.

> Limitations

> Works best with frontal face views and moderate head rotations.

> Extreme head poses or gaze directions may produce less accurate results.

There it is. To use this I’d like to see an example showing it stop adjusting when “extreme” aka normal head poses are used. If it can handle real behavior and improve eye tracking in the optimal case so it’s seamless adjusting / not as someone moves around, that would be a good product.

technically cool, however I'd rather prefer some semi transparent mirror set up.

such a set up keeps the eyes alive.

I do appreciate that this is a problem worth solving though, and I spent a lot of my time during COVID worrying about the negative impact that normalising loss of eye contact would have on the social interactions of our younger generations.

Back in 2021, I took one of those £50 teleprompter mirrors that YouTubers use, put a 7in raspberry pi display in the slot where you're meant to put your phone, and made it my 'work calls display' for a couple of days. The interesting thing is that the only people that noticed without me pointing it out were completely non-technical, and when they did they complemented me on the quality of my webcam rather than the fact I was looking straight at them; they could tell something was better, but couldn't quite put their finger on it. Which is funny because I'm sure being stuck behind a cheap perspex one way mirror made my actual camera quality a bit worse.

I remember I got to the point where I started playing with cv2 trying to do realtime facial landmark detection on the incoming feed and having a helper process shift the incoming video window around the little screen so that it would keep the bridge of the other person's nose (the point I naturally made eye contact with) pinned to the bit of the screen that was directly in front of the webcam lens. Then one morning I walked into my office, saw this monstrosity on my desk, realised I was nerd sniping myself and gave up.

One thing I do remember though is how odd it felt looking at yourself in a mirror without your image being mirrored. Not sure my brain was ready for that one after thousands of years of looking at itself in mirrored surfaces.

Bit of a weird pic but the only one I can find: https://pasteboard.co/BXE6zhbpOD7E.jpg

Several years ago during the pandemic, I enlisted a job coach to get me hired. One of her paramount concerns was my eye-contact with the camera. She said it's so important. Am I paying attention? Am I an honorable man who maintains eye contact when I'm in a conversation? If I look away, am I collecting my thoughts, or prevaricating?

Many supervisors, managers, and teachers will judge their employees by whether they can pay attention during meetings, or if they're distracted, in their phone's screen, looking at keyboard, glancing off at children or spouse. Even more important, if you're meeting your wife and she can't even maintain your attention, what kind of husband are you?

If you employ a gadget to lie about this, then I hope they fire you and find someone who'll be honest. I hope your wife sends you to sleep on the sofa.