Show HN: Unscreen – Automatically remove video backgrounds with ML | Hacker News Reader

Show HN: Unscreen – Automatically remove video backgrounds with ML

unscreen.com

215 pointsgroe107 comments

This has been submitted multiple times as a “Show HN” but as it is a paid service, I guess this is a marketing strategy.

https://news.ycombinator.com/from?site=unscreen.com

Doesn't this violate the spirit of Show HN? I thought only the authors could submit them unless they are all authors of the project.

If you want to get traction, you need to be far more creative in your business model. The main application of background removal is online meetings and its already well implemented but it leaves a lot to be desired. One thing you can do is to make service completely free. Offer a desktop app that creates a virtual camera that you can use in Zoom, Teams etc. Differentiate yourself by adding features such as much more cool and fun background wall papers or even live videos. Focus on adoption, not revenue. Once you get traction, offer premium background images or videos. Create a marketplace where people can buy/sell. Focus on buyout by big players because ultimately they have the platforms with millions of users where you can provide far more value.

Removal of background in images is also big (your site only seem to offer video/gifs). I've personally tried half dozen solutions and all fell short in low lighting scenario. If you are actually doing good here, you can beat competition. There are probably hundred websites out there for background removal for images, none works well.

Sounds like that would evolve to become Google AdSense for your home office.

According to the FAQ this only works with prerecorded video, not live video.

> Once you get traction, offer premium background images or videos. Create a marketplace where people can buy/sell.

A lot of this seems like guesswork, and honestly also like terrible advice. Maybe if the person had added their experience or a story about their own experience. Like the other person wrote, some advice might be irrelevant because this person hasn’t understood the product to know it’s it’s for existing footage only, not live footage.

Sometimes advice can be useful, but be wary when it sounds useful at first glance but includes no examples, and no credentials.

Then often it’s someone who considers themselves a success guru, without much sincerity. They just think that their advice is obvious, when its probably outside their domain. It’s not bad faith, it seems to me they’re just a wannabe startup guru/investor/founder/whisperer (what I write here might also be crap).

The best advice I’ve found comes with intimate stories of failure. Stories that help you understand the underlying friction, yet aren’t specific to the exact challenges you’re facing, but leave you with solid hypotheses to explore.

There is a lot of negativity in your response without even deluding to what's really wrong with the advice. There are tons of very successful founders recommending focusing on adoption instead of revenue in initial period. Freemium is very well adopted all over. Literally entire app industry worth billions of dollars use this model. Again, I'm not claiming to be expert in this domain. You are entitled to your opinion and frankly we are all armchair pundits here, however, it's bad form to spew out so much negativity without any reasoning or even alternative.

Yeah I understand where you’re coming from, and maybe my reply broke my own rule of giving advice without a personal story.

Where i’m coming from is having listened to advice from people who I looked up to, but who did it to get rid of me (because I was ignorant of protocols/etiquette), rather than being genuinely interested in my project. I’ve learned that the SV world has a high rate of failure and many are hungry, and they try to also carve out a role for themselves, or try to get involved somehow, even if that means they’ll claim competence in an area they’re not competent in. I was more naive then, compared to today, but believing these ‘gurus’ was a painful learning experience for me, because their advice often left me stranded in the end.

I’m not negative, it’s more just sharing my disappointments and struggles. Thanks again for your comment, which made me see I kinda broke my own rule

> Again, I'm not claiming to be expert in this domain.

Actually nowhere in your original post do you include this "not claiming to be an expert" disclaimer.

As a result, when I read your original post, I assumed that perhaps you were an expert as there was no self-doubt at all evident.

I think perhaps that's what the allegedly negative post that you are responding to is getting at.

>There are tons of very successful founders recommending focusing on adoption instead of revenue in initial period.

As of 2020, this is the exception rather than the rule. The problem is that focussing on adoption before focussing on value is very risky. You are constantly flirting with premature scaling before having found product/market fit (see e.g. Startup Genome project).

However, successful businesses that have been built on adoption are by-design much more likely to be well known. Again, it's entirely possible to build a business that relies on network effects or economies of scale, but it's not the default advice I would give to founder.

>Freemium is very well adopted all over. Literally entire app industry worth billions of dollars use this model.

No, you are confusing freemium with advertising here, which is the predominant business model. The challenge with freemium is the conversion into paid, which you cannot predict unless you have proven the value hypothesis already.

The main application of background removal is online meetings and its already well implemented but it leaves a lot to be desired.

That's the most obvious case people see right now because we're doing more video calls in lockdown and people see it as a funny gimmick to play with on Google Meet, but background removal has been around for years in the professional video and film industry. Video chat is absolutely not the main application of the technology.

The video chat market though would be approximately a gazillion times larger than the professional video and film industry wouldn't it?

I have no idea. There's a gazillion times more people doing video calls than make professional videos, but not all of them would be willing to pay for the software and those who are probably wouldn't pay very much. It's the age old question of whether or not it's better to sell something cheaply at high volumes or more expensively at lower volumes.

Good webcams are hard to find at the moment because people are very willing to spend money on their video calls.

> people are very willing to spend money on their video calls.

MORE people are willing to spend money on webcams than in the recent past. That is why there is a supply issue. This doesn’t necessarily mean a huge number of people are willing to spend money in it in general. Doubling a relatively small number doesn’t automatically make it a big number.

>the age old question of whether or not it's better to sell something cheaply at high volumes or more expensively at lower volumes.

Factor in the support you need to accommodate your customers and you have your answer :P

You can just ignore support entirely and then it costs nothing. It's what Google does.

Better yet, create a "Help Center," that has 1,900 articles that all have a link that says, "Need more help?" and then that link goes to another of the 1,900 articles that also has a link that says, "Need more help?" And that link goes to another article, which has a popup that reads: Call 1-800-ABCDEFG. Then they call that number and get an automated system that says, "Thank you for calling. How can I help you today? You can say things like, 'I need help,' and 'I need different help.'" And then they have to speak all the menu choices. And the 0 button doesn't take them to a representative, only back to the first prompt.

This is too much fun to brainstorm.

Pretty hard to compete with the free implementations in most video chat tools...

What about marketing to live streamers and youtubers

Surveillance video processing and intelligence services are probably by far the largest use cases for video background removal and similar techniques. Far smaller in terms of number of customers, but far far larger in terms of revenue specifically for this capability.

Millions of people paying peanuts (attributable to this feature) is way less than a few hundred or thousand research / defense labs, contractors, etc., paying tens of thousands up to even millions on licensed software for this type of thing.

In both teleconferencing and surveillance you are dealing with static background, no need for fancy ML algorithms.

No, not true. Surveillance cameras mounted on police cars, helicopters, etc. Even in stationary cameras, scene understanding is a hard problem. Suppose someone wheels a dolly of boxes into a warehouse and leaves them. Are they now part of the background? Are they suspicious boxes? The coarse level state transition of objects in the scene is really hard to solve even with heavy ML.

There is a reason background removal sucks on chat apps. I fiddle around with the state of the art approaches, it takes a very expensive machine to get to an acceptable frame rate. It wouldn’t scale.

Would this be a good candidate for P2P? As what if your computer GPU could be used by other app's users to do a small part of such heavy operation? (when your GPU is not doing anything else)

Latency.

Can you elaborate more? What are the approaches?

Not OP, and not in this field, so do some more research, I am just sharing what I found:

https://paperswithcode.com/task/video-background-subtraction

I'm pretty happy with XSplit VCam (https://www.xsplit.com/vcam)

Anything like this for macOS?

Pre-existing example: Snapchat has an app to apply filters in webcams

The quality level here is way beyond what Zoom has or needs. This is a professional tool for creative pros — knocking out an image background is too easy but video is a real challenge. And this kind of work at this resolution would be prohibitively expensive to do for live video ... you’d need a Titan RTX or similar.

> I've personally tried half dozen solutions and all fell short in low lighting scenario.

Have you tried https://www.remove.bg ? I came across that site a while ago and was surprised how good the results were that I got. YMMV, though.

[EDIT]: Looks like Unscreen is actually from the same people as remove.bg. [0]

[0] https://news.ycombinator.com/item?id=24058701

We've built Unscreen on top of the tech of remove.bg, at least partly, and the #1 design goal for both products is a high result quality. We've summarized a few of the improvements recently here, if you're curious: https://www.remove.bg/b/introducing-remove-bg-x2

remove bg/unscreen are pretty much cool ML applications looking for a market

This looks like it's targeting pros like me who need to remove background from high-quality videos. The price is completely OK.

My rule of thumb for advice on the Internet is that if it is mostly in the form of statements like "You need to do X, Y and Z" and has very few questions intended to dig deeper into the nuances of the product and business model... then it's not generally worth much at face value.

Essentially: The less information a person needs to dish out advice, the less value I am going to place on their advice.

haha you said what i wanted to say in a way more concise way. nice one

> Offer a desktop app that creates a virtual camera that you can use in Zoom, Teams etc. Differentiate yourself by adding features such as much more cool and fun background wall papers or even live videos.

https://www.xsplit.com/vcam does exactly that already and is available on demand or for a lifetime license of $40. It works mostly ok. Unscreen's quality seems a lot better though.

Reading “The Lean Startup” by Eric Ries.

He argues that this kind of “build it; they will come” aka “Just Do It” business process is a recipe for failure because you make a lot of untested assumptions about your market, and front-load a ton of (often wasteful) work.

He suggests the “Build, measure, learn” feedback loop, which you actually plan backwards:

1. Figure out what you need to learn by outlining the key assumptions, including your value hypothesis and growth hypothesis.

2. Decide how you will measure your success in validating (or invalidating!) those assumptions with science (eg he calls his method “innovation accounting”).

3. Build the smallest experiments (or MVPs) that will allow you to learn the things you need to learn.

4. Make adjustments to your hypothesis, rinse, and repeat.

Another common theme, “get out of the building” and actually speak with all kinds of potential customers. Since a start-up operates in an environment of high uncertainty they cannot rely on traditional planning.

Anyway I love reading the book and learning about the early days of Intuit, Dropbox, etc and how sometimes the way to make the most progress feels very counter-intuitive.

It almost feels like a real world version of Stochastic Gradient Descent for businesses to mechanically gravitate to various local minima on an invisible failure landscape.

The price and business model is perfectly fine:

1) most pros use green screens for background removal

2) for (rare) shots where background needs to be removed and green screen cannot be applied, a high quality removal like the one here is needed

3) for such rare occurrences, charging higher amount of money is completely appropriate and no reasonable pro would complain as it would save them additional costs of re-shooting the scene completely or doing the removal manually

4) nobody has anything similar, so they can charge premium while it's possible

Given what I mentioned I'd say their price is too low anyway.

Also recommend "The Mom Test".

Right, this seems like a feature, not a product -- and it's not difficult to replicate. ML is becoming more and more of a commodity these days.

The very first thing I thought when I saw this wasn't "ooh, cool tech for voice chat", it was "man, the special effects industry will KILL for this - no more manually going through every frame of an effects shot and painstakingly removing background elements!"

So basically, I think the main assumption on which you built the rest of your advice is flawed.

I don’t think it’s feasible to do this level of processing in real-time _for free_.

From same authors that make tool for removing backgrounds from images - remove.bg

It was discussed here: https://news.ycombinator.com/item?id=18697601

BEWARE. remove.bg seems to have a malware/fake Chrome update popup.

Fresh Chromium installation, no adblocking or other content blocking plugins. Zero advertisements and popups on that site. I keep reading these kind of comments on HN and everytime it seems to be only that one person reporting the problem that sees that behavior. Are you sure your network is not compromised?

Perhaps the GP is receiving malware through a targeted or non-targeted advertising campaign.

> Are you sure your network is not compromised?

It's definitely not. Also used 3 different browsers - each one showed the same dialog but with the name and version of that browser. It doesn't happen on my mobile over WiFi (rules out network injection).

This is what I'm seeing: https://imgur.com/a/o9PNmaU

It's being inserted via a .js file on static.remove.bg:

https://static.remove.bg/remove-bg-web/38c6be57b031c26a2186b...

(search for 'out of date' )

So either the site owner is complicit, or they've been hacked.

Edit: Fired up a VM of Windows 7 - Same message - so unless my routers been hacked to inject that script somehow, i'm 99% certain it's not me.

Are you sure it's malware? Looking at the code it seems to redirect to "browser-update.org" and is genuinely redirecting only older browsers based on unsupported features.

Looking at the code, it is using a script from Browser Update[1] to determine whether to show that message.

Here's the line from the js you linked to that checks the browser version:

    window.checkBrowserVersion = function() { i()({ required: { i: 12, e: -4, f: -3, o: -3, s: -1, c: -3 }, insecure: !0, unsupported: !0, reminder: 0, reminderClosed: 168, style: "corner", api: 2019.06, test: !1, onshow: function() { window.track && window.track("BrowserUpdate", "outdated_version_dialog", "Browser version outdated dialog") } }) }, $((function() { window.checkBrowserVersion() }));

The c: -3 is the crucial bit that should cause it to trigger if your version is at least 3 versions out of date.

Yours isn't though. Can you access http://browser-update.org ? If not, then there might be something with your DNS settings. Have you tried tethering through your phone? Have you changed your user agent?

1. http://browser-update.org/#install

Just checked, and my Chrome is on version 84. Version 80 is out of date.

You need to update Chrome.

To be frank, I don't need to do anything. I'll update Chrome when I'm ready, thanks.

What I object to, is a dodgy popup appearing on an unrelated website.

That's interesting. Number of versions is probably a weak signal for this --- but how many browser versions should the dev a small site be looking for? Things DO break...

You don't need to call an ambulance when you break your leg, but your browser has at least five high vulnerabilities (and one critical vulnerability).

It's a hint for outdated browsers, it's not an ad and there's nothing malicious behind it (unless there's an issue in your local network or machine perhaps?).

http://browser-update.org/ https://www.npmjs.com/package/browser-update

It's a fine line though to not get into the way with things like this, and we could certainly improve it's appearance a bit - sorry for that.

I think trying to emulate a built in browser notification is the wrong way to go. I wouldn't have got as concerned if the popup looked more like a web page overlay, with wording along the lines of:

"Hi we're from Browser-update.org (clickable), and this site has asked us to check your browser version. Your browser version is out of date, and should be upgraded. Please go to your [settings] menu to update, or visit [us] for more information."

Of course, feel free to totally ignore my suggestions - it's a free world. ;)

I'm curious if this is just repackaging existing OSS algorithms for isolating foregrounds, or if they have an actually new implementation that outperforms them

It's actually quite funny that the Zoom auto remove background does a pretty good job. I can't get the same results easily in Adobe Premiere actually.

Snapchat filters are even more impressive and it is magnitudes more difficult and time consuming to get similar results in After Effects.

I really wondered if I was missing something so watched a lot of YouTube tutorials on background removal and object tracking and all I see is that it is very laborious and even than not always a good enough results with edge/transparency artifacts etc.

Adobe really needs to implement new features for this because the expectations are changed by these apps and the bar is much higher.

In my experience zoom does a horrible job. It's a flickering mess, half the head is missing, etc. Perhaps in great lighting conditions it's better.

IMO Zoom‘s background removal is not good at all. It‘s surely way lighter CPU-wise.

Looks very impressive, but it seems cripplingly expensive.

I don't play with this much, but when I use my green screen, I'm often doing 15 minute videos, so it is well out of my hobby budget.

The application of this would probably be different from what your use case is (since you can use green screen).

Most kinds of professional applications I could think of would better off with a green screen. On the other hand, using this for editing videos in the wild with messy background still not good enough.

Looks cool, this model is start of the art I believe https://app.wandb.ai/stacey/greenscreen/reports/Two-Shots-to...

I believe this is homegrow from their photo background-removal tool remove.bg

Probably frame by frame, not real-time. But not less impressive.

Looks very impressive, but please do not autoplay all the videos at once on the website. My computer fans started to spin like crazy!

It killed my Android tablet running Firefox! First everything slowed down, then the status bar disappeared, then... black screen. Had to restart.

I'm genuinely surprised my Pixel 3 phone had no problems with it in Chrome. That was a lot of video.

Video background removal, when hair is involved, is painful. Clearly this is not targeted at webcam footage. Anyone who has had to manually remove a background using a tool like Premiere or After Effects from pre-recorded video would see the benefit of this tool/api. It is pricey but i can see motion graphics studios using this regularly.

€ 1.98 / minute - wow.

It's for 30 FPS. And only up to 1080p.

> Up to 30 fps: Base price

> 30-60 fps: Base price × 2

> 60-90 fps: Base price × 3

That pricing looks very much "what it costs to run the service, plus some profit margin".

A service run by anyone with any business acumen will charge what customers will pay, modified for benefits that customer brings the company (like referrals, bigger revenue leading to a higher company valuation, etc.).

None of that depends on the frame rate or resolution.

Video processing tools in the cloud very regularly use frame rate and resolution as pricing inputs, similar to GB with storage solutions

I'm guessing the OP aims at professional, high-quality video editing rather than at a hobby use. Which is probably the right thing to do as there are low-quality free options already. Hard to compete with that.

Wouldn't somebody doing professional video rather use a green screen instead, though?

I think some of the artifacts on the video at 0:12 (handbag thing) are very noticeable. Same thing at 0:19 (cat)and at 0:28 (cat again). Otherwise, looks nice :)

Some scenes, foregrounds, backgrounds, lighting conditions, movements, etc work better than others - there's a lot of potential for errors in videos and the videos from the examples page are straight from Unscreen Pro. We're continuously working on improving the AI though (NB: We don't use video uploads for that - see unscreen.com/privacy)

This is sort of the inverse of gimp's "foreground select tool". It was a PHD thesis on the algorithm to select foregrounds that was contributed to the gimp photo editor as open source.

https://docs.gimp.org/en/gimp-tool-foreground-select.html

And the paper with an overview of the algorithm is here: "Image segementation by uniform color clustering"

https://www.inf.fu-berlin.de/inst/pubs/tr-b-05-07.pdf

This is an area of research called natural image matting that has been going on for over 20 years.

Some influential papers include:

https://www.microsoft.com/en-us/research/wp-content/uploads/...

https://webee.technion.ac.il/people/anat.levin/papers/Mattin...

I feel like this could be presented in a much more engaging manner.

There's one quick video on the main page which shows replacement backgrounds, but the "Examples" page shows about 15 or 16 examples of the same thing, removing a background. It quickly gets to the point of "Ok yeah, so what? Show me something cool to do once the background is gone."

I get that it may be difficult to remove a background the way this service does, but the customer is going to be less impressed with the fact that it works correctly than they are going to be about it having an actual engaging or useful purpose.

You can try it with an uploaded video or GIF here: https://www.unscreen.com/upload

That's a great idea, thank you

Since it doesn't do live video so no chat background virtual camera tools - but what if you did a video in the area you wanted to remove could you then use that for some sort of baseline for removing the background? So you do a video in the area you want to use your virtual camera in, and then when you're live it knows what the actual 'background' looks like so it can be used to remove live.

If you can keep the camera at a fixed position, that’s surely a great option. I think that’s not the use case this tool is for. A green screen would also work obv.

I want to integrate your service to remove background for product images. However, not everything works well, especially when background and foreground color are close.

that's fine. The thing is that on the website, there is a tool to fix imperfect results, but with the api, there is no such tool.

Since you mention images I assume you are referring to remove.bg, not unscreen.

We're working on an SDK that will allow for editing on external sites. You can sign up to our mailing list if you want to get notified when it's ready: https://www.remove.bg/blog#subscribe

Thanks a lot for your comments and feedback - some great great advice and ideas in here!

This is the MVP release of Unscreen Pro, so there's many more things to come in the future, from quality improvements, to an API, integrations, editing tools and more. Appreciate all the inputs :)

Looking really cool, I just hope a Premiere / After Effects Update won't kill your business.

At the prices they're charging, they're rather asking for it.

Very impressive results and great presentation.

I imagine this can be useful for business that want to record small 360 of their products, but not invest in a big Physical setup.

Thanks! Yes, professional 360° product shots are way too hard to produce right now and we're aiming make them much more accessible, particularly for smaller vendors

the autoplaying movies are massacring my crappy cell internet connection

(my home is currently one of the hundreds of thousands without power on the US east coast)

Set uBlock Origin to block media elements larger than 50KB or whatever threshold you want. More info: https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-...

Looks great and is exactly what one needs for professional screencasts if she doesn't want to fiddle around with a greenscreen. Thumbs up!

Looks Great! How tool is built?using computer vision,image processing?

It's a combination of deep learning and traditional computer vision algorithms. We do a lot of basic research, evaluate many different approaches, train them in dozens of variants and ultimately combine what performs best, then try to make it as easy to use as possible

Whats the tech stack this was built on?

Somehow I parsed the title as next-gen adblock. Well, it is great to see the required technology is already here.

How long before this is used to help fake political videos from existing footage. Sigh

Need longer demos to show temporal contiguity.

I think processing is done frame per frame.

Product hunt is full of this kind of stuff, is there anything different on this project?

$1.98 per MINUTE?

I thought I'd woken up in 1980.

But I guess this is aimed at people making professional video presentations etc rather than for webcam meetings.

It seems to work pretty well, but none of the examples have any sudden movement, which is where there's often tearing etc.

It works on pre-recorded video so webcam meetings are not their target market.

Exactly as I concluded, and said above.

Reply on news.ycombinator.com