OBS Plugin: Background Removal | Hacker News Reader

OBS Plugin: Background Removal

github.com

149 pointslnyan59 comments

Given how severely imperfect this approach is, I’d much rather people didn’t use it: it’s disconcerting and distracting. Either do it properly (which tends to mean chroma keying, preferably with thoughtful handling of lighting too if you do anything like background replacement rather than just removal), or don’t do it at all.

I have not the faintest clue whether this position of mine is popular or unpopular.

Better models may improve matters (the one shown here looks to be decidedly worse than some), but even the best I’ve observed at this time is still something I’d prefer not to be used, because it’s still wrong too much of the time in annoying ways.

100%. My new boss both has a background that is 'our office open plan area but empty' (he must have taken a photo when the place actually was empty) when he's at home. Plus he now has some greyscale or sepia filter so that he looks like a ye olde ghost for some reason. Not to mention he has headphones where you see these slivers of no background through them around his head.

I asked him if he was using some sort of filter and he was like "I don't like the color of my face in my camera". He stands out like a sore thumb in the grid of people in video in larger meetings now too as the only one not in color and with a weird background (others just blur it which I also kind of hate - but less).

I find calls with him with the video so distracting. I would much rather see his living room or whatever he's trying to hide - and it being real - than this weird fake office thing he is trying to do instead...

Maybe it is a sign of privilege though that I have a spare bedroom with a white wall behind me I can use without background or blur. I remind myself not everybody is so lucky...

I asked him if he was using some sort of filter and he was like "I don't like the color of my face in my camera."

If he was to use OBS with the virtual camera as a middleware between him and the call he could apply a LUT (lookup table) filter to the camera output to correct the color[1]. I do this when I'm presenting to a lot of people or when a call is going to be recorded. It makes a heck of a difference to the quality.

[1] An explainer https://streamshark.io/obs-guide/color-grading-lut.

> My new boss both has a background that is 'our office open plan area but empty' [...]

During the first lockdown, a friend of mine sourced a photo of the inside of his local (closed-down) pub, and he used that as a background.

"Working from home? Of course I am. Honest..."

I've done that. In my office I have a 27" Mac from 2011. I keep it around as a 32-bit machine, and I decided one day to try using it as a video conferencing computer, because it has the biggest screen out of everything I have. The camera, on its default settings, makes me look like I have a skin disease. It's really horrible. But Macs from that era could have their video stream intercepted and tweaked by third party systems, so I desaturated it to black-and-white.

The first paragraph of your comment is so Dilbert I laugh out loud every time I read it

I don't understand what people are so eager to hide in their cameras

Unsightly and messy background.

In my case, location. It's nice to be able to join a Zoom meeting from a Starbucks or a restaurant.

Usually my socks hanging up on the drier behind me.

We’re gonna talk about this on Monday, [name redacted].

I was watching some court proceedings and one of the parties had the background blur on and every so often when the person half turned their head to review a document nearly half of their face disappeared under the blur. It was very distracting.

Perhaps if people were a bit more professional and stopped commenting on other people's personal space that's taking up a fraction of the screen real estate we wouldn't need features like these.

My boss can't help himself making smart ass comments about my background, at one point I managed to get a plain white wall and even that scored a comment.

Your boss sounds immature, but furthermore why do you care?

Wait, why do people care about other people's personal space? It's just houses and offices, what's there to hide about that?

I have an array of RC planes on the wall behind me (because they take up room and mounting them vertically on the wall was a good solution to save space, they make for a great background.

I wish we would stop using the "what's there to hide" argument. Let's just give people a right for default privacy without questioning what they want to use it for.

That's why nobody here is questioning whether people should be able to use the blur filter.

Why use video as the default in the first place?

I would consider changing your boss. What a piece of work. My guess is he’s flawed in many areas…

> My guess is he’s flawed in many areas…

That's a dead giveaway of being human.

Chroma keying requires a green (or other solid, "unusual" color) screen, though, right? Most people don't have those, and won't want to expend the effort to set something like that up. It's certainly imperfect, but it's useful for people who'd prefer to obscure their surroundings while on a video call without having to deal with a more "professional" setup.

The problem is that those people don't grasp how ridiculously awful it looks. Do you pay money for decent clothes and a decent haircut? Then spend the money on either a greenscreen, or anything else to put behind you to cover whatever you want to cover without making you look like a special effect done by a third grader.

I pay about $30 every 3-4 months for a haircut and most of my clothes are 10+ years old (one of my favourite shirts, that I regularly wear in the home office, is the Australian team's football jersey from the 2006 World Cup).

I cheerfully use Google Meet with it's poor blurring that constantly pops things in and out of the background without giving it a second thought.

I judge people by the content of their words and the merit of their actions, not what they look like or how much effort they've gone to in order to have a perfect webcam production.

This tool looks super awesome and I can absolutely see myself using it. I've tried greenscreen in OBS before and decided it wasn't worth the effort for my purposes; something that is roughly decent as this looks at a glance will be really useful.

Blurring the background is different than putting some other sort of background there. That doesn't bother me nearly as much.

Regardless, I am someone who never pays for a haircut and has very casual clothes. But this is something that, to me, indicates a complete lack of sensitivity to aesthetics, and in a large number of jobs/roles, that's a really bad sign.

There's also a bit of the "what is so important to hide?" question.

It's great if you can ignore such things, but not everyone can or does. This is almost like someone who has poor hygiene, who has dirty hair or a bit of a smell to them. Anyone would advise such a person to put in a bit of effort if they want career success. Same thing.

Nah, the blurring is just as bad. In the best of cases I find it to be firmly in the uncanny valley, as I also do of what little I’ve seen of software-based depth of field manipulation or artificial blurring in smartphone photography. In the typical case, it suffers from exactly the same issues as background replacement because it’s doing exactly the same thing, estimating a mask for the areas of interest and altering the rest. Half the hair will be blurred one moment, then one shoulder, then half the chair will flicker between being blurred and not blurred—it’s just as disconcerting and unpleasant.

> There's also a bit of the "what is so important to hide?" question.

It's a principal thing for me. My space is mine and work doesn't have a right to see into that. They aren't welcome in my home. Maybe I have a good office setup or maybe not. Maybe my home is clean today or not. It wasn't anybody's business pre-covid, and it remains nobody's business today.

Is it so hard to have just one portion of wall with a neutral color in one's home office setup?

I work in different part of my home, but I dedicate one small space for video calls, where I stand, not sit. It is designed to provide a plain background. If I didn't have a wall at my disposal I would have installed a white ikea roller window blind on the ceiling. It is much more effective than any blurring/background replacement algorithm.

You choose to work from home, you choose to welcome work in your home. The good thing is you still have the liberty to limit and design the small part of the home you are welcoming your coworker's in.

Assuming that you have a home office setup. Me, I have been working for 2 years from my dining table.

And people might see that as that you are working from home, but not willing or able to allocate a dedicated space for it so you can actually be as effective as possible. Nor are you willing to purchase or make a simple backdrop.

Which seems to be accurate, and in my opinion rightfully creates a negative impression in their minds.

A green screen also doesn't have to be expensive. Go to a fabric shop with a laptop running OBS (or maybe even a mobile phone that has an app that can do chroma keying), and find a roll of fabric with a color that works well. You should be able to find something that works, and it should not be more than one haircut worth to buy a few square meters of that. I did this myself at the start of the pandemic to be able to record some lectures at home, and it works just as well as a professional green screen (which I got later, its main advantage being easier to set up).

How did you get your job to pay for for the fabric and the time to set this all up?

I didn't. While the lectures were related to my job, I wanted to be able to keep using this equipment privately, and in such cases I prefer paying for it myself to avoid any conflicts of interest and the possibility of me having to return the equipment.

That assumes you have the space to put something behind you like a green screen.

Oh come on stop pretending it's so easy to setup a chromakey with most video chat applications. The only ones that don't scream about virtual cameras are Zoom and with the last updates MS Teams the rest just doesn't work. Ofc you can use NDI but than you have to setup like 3 programs just to get an effect that is available on most plattforms with just a click even if the quality is worse.

I just do it in OBS and expose the result as a virtual camera. Zoom/Slack/Meet just use the virtual camera with no additional config

Just for another point of anecdata, I've also had success with OBS' virtual camera with most video conference apps, besides specifically Jitsi in Firefox, which seems to have an issue where it forces loading the hardware camera before letting you choose the virtual camera, and that fails because OBS is already hooking it.

You are one bed sheet away from chroma keying in OBS. Since you can easily configure the color to be keyed, you will have luck with anything besides the color of your skin or your eyes. I tested it with a green sheet fitted on the bookshelf behind me and since bed sheets wrap around roughly the size of it, I did not even need any tape or clamps to fit it.

I agree this approach can be distracting, but I'm still glad OBS is adding this feature. There are times when it's "better than nothing" (if you're in a coffee shop or an airport or something where your background would be genuinely distracting). It's important to remember that OBS is a bit of a "swiss army knife" and there are plenty of use cases where this could be handy.

That said, I've done background replacement using chroma key with OBS and it's quite easy. Anybody serious about streaming (or recording) video who wants to remove the background should own one. I own something called the "web around" which is a fabric disc supported by a tension wire around its circumference that attaches to the back of your chair. I'm convinced that this (and a decent microphone) do make a difference in how people perceive me in meetings and presentations.

I also find it distracting and prefer people wouldn't do it. However they have the right to privacy and not showing me their house, so the choice is theirs.

The implementation matters a lot too. I found that for most people the Zoom implementation is pretty bad. The Google Meet one is typically less distracting especially on blur mode. (because the flickering isn't as noticable than when flipping between two completely different colours)

My company had the marketing department design a few abstract backgrounds with company colors / logos, which look fine on their own, but incredibly weird when there's a blurry and badly cut-out person with a window behind them. Basically it looks like there's a challenger approaching in Smash Bros. We only have to use them when talking to people outside the company, but most people just leave them on all the time.

Personally I'm not a fan of background removal/replacement unless it's done properly, but I do like blurring. I use a good webcam and I mounted a soldering light above my screen as a ringlight which works surprisingly well. Plus nvidia broadcast for noise filtering and background blurring, which makes the task easier for teams to blur it some more without cutting into me. Overall I think it looks fine. My home office looks too messy and unprofessional when in focus, but has nice colors when blurred heavily.

My boss often sits in his closet with lots of suits and shirts behind him, which confuses the filter enough to replace all his hair with the company background.

I think people should just decide for themselves what to do and just learn the basics of how to do lighting. Nothing fancy, just make sure the light comes from the front, not from the back, use a proper webcam (or maybe your phone, that might have a better camera+processing), see if you can do something about your background to make it presentable or easier to blur. Of course some people don't have the luxury of a separate room in their house for nothing but work and the money to turn it into a movie set. Maybe they are forced to work in their bedroom / kitchen / closet and don't want everyone to see how they live. I think in that case background replacement is the lesser evil.

I won't use the blur or fake backgrounds normally, I find them goofy. I just let people see what is behind me. If it not appropriate for video I'll leave it off. My background has been, the living room, kitchen pantry, bedroom, forest behind my house, patio, whatever. We are real people, in real locations just show it.

the primary usecase of OBS is for twitch/content creation streaming i think and for that use-case it's pretty good. however as the system resource usage is pretty high i don't think this will see adoption for much compared to chroma keying.

Google/Mediapipe has a better performing models here https://github.com/google/mediapipe/tree/master/mediapipe/mo...

I can confirm this with a couple of Codepen demos. MediaPipe's selfie-segmentation model[1] seems to give much better results than TensorFlow's body-pix model[2] (which is the model I most often see talked about for this sort of background removal work).

As much as I admire ML models (and I remain in awe of people who know how to build them), I still think there's room for more conventional chroma key filtering - for instance identifying part of a video's moving display so you can add some UI to it[3].

[1] - MediaPipe demo - https://codepen.io/kaliedarik/pen/PopBxBM

[2] - TensorFlow demo - https://codepen.io/kaliedarik/pen/ZEeoZaP

[3] - https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html

This is awesome. The last time I looked, the only background removal I found for OBS required a green screen.

It's a shame Nvidia broadcast doesn't work on Linux. If you use this with CUDA, are the results comparable?

I plan to switch to Linux for my main machine and the only thing I'll miss is Nvidia Broadcast. I know the voice part of it is integrated in OBS, but not sure if that works on OBS Linux.

RNNoise works good enough for me

Seconding that RNNoise works very well and ships with OBS. I do however not use it as I find it to be a bit too aggressive and close to a noise gate at times. But I fully recognise that falling back on an analogue expander is not something everyone can do.

I use Nvidia Broadcast, it has the least artifacts and the roll of when blurring is also the best I think, downside it uses a lot of GPU and my RTX2080 fan spins up.

It works surprisingly well! I started using it for the noise filtering, which is even better.

Are you using a lot of the other filters too? I use Broadcast's background removal and the tracking zoom on a Quadro 4000 mobile and it remains mostly silent.

Why hasn't anyone built a plugin that says "sit still, now move left, now move right" and then just remove all the pixels that didn't move? Seems like it would be easy and work as long as the camera doesn't move.

Is this a harder problem than I think it is?

I think current algorithms do a version of that, but it has to be coupled together with person/outline recognition and such. Taking the background in just once doesn't work because the pixels still change over time (due to lighting and such).

What's really impressive to me is that it looks like web apps like Google Meet and Discord seem to do this in JavaScript on the fly.

My kids missed out on most of the remote schooling thing due to location but they got 8 days earlier this year and made the most of it setting up OBS under linux with this plugin and fed it into MS Teams as a virtual camera.

They made up several scenes they could switch between with subtle animated backgrounds and added chroma keyed meme/animated stuff they could enable during less formal time like end of lessons.

It isn't as good as the closed source offerings from big companies but those companies don't have enough users on Linux to justify developing for the platform so it is a chicken and egg thing. For what it is I was happy with it. With a bit of adjustment it worked suprisingly well in actual use. One of the kids was stuck with an old under powered laptop with integrated graphics and it had no problems at all running this.

If you want to do this stuff properly you obviously need a green screen and good lighting but when you suddenly have an entire family home and online and are trying to juggle access to computers, quickly locating and finding space for multiple green screens and studio lighting isn't an option for regular people.

So if a proper chromakey setup is more of a commitment/hassle than many people are willing to accept, and software background removal is imperfect to the point of distraction, why aren't depth-sensing solutions (like the $300 Intel RealSense D435) an ideal solution to this problem? What am I missing?

This is awesome! I made a while ago a cmd line background remover using the same model, https://github.com/nadermx/backgroundremover

This looks cool, being both free and cross platform.

A non free (4 hours free a month) solution I quite like on macOS is krisp.ai which I also use for noise removal. Background replacement was recently added to M1 chips.

On the Windows side, Nvidia Broadcast has worked well for me (requires an RTX card), it also supports tracking which I really like. I've previously tried XSplit but the results were worse than whatever implementation Zoom and even Hangouts in a browser use.

I rarely video call under Linux (non existent laptop drivers for my webcam) but look forward to trying this out if the need ever arises.

Finally! So glad to see a plugin make an earnest attempt on this now. Been using Snap Camera green screen background combined with OBS chroma keying to make it work, but will try this out now.

This is a great plugin. I'm using this to have background blurring with the Linux MS Teams client, which for whatever reason does not support this natively.

"AI". Wow this looks bad, worse than just taking a screenshot of the background and subtracting from video feed.

Reply on news.ycombinator.com