Meta Releases New AI-Based Photo Segmentation Tool to Everybody | Hacker News Reader

Meta Releases New AI-Based Photo Segmentation Tool to Everybody

arstechnica.com

204 pointsBrajeshwar76 comments

Discussion from yesterday: https://news.ycombinator.com/item?id=35455566

I relly hope Meta (FB research) releases this wonderful noise reduction library: https://github.com/facebookresearch/denoiser

I tested MANY while building various audio tech and this was by far the best (beats the shit out of all the python Pandas Hugging Face lists etc). Incredibly ability to cut noise (tho, it must be noted that perceptual improvements for humans do not usually increase machine transcription as AI models strangely seem to pull info out of the dead space between words, and the noise around words...not just the voiced words themselves...the hidden vibrations, beyond our human ken, oh mere mortals unworthy of the grand perceptive machines...ugh...:p :o ;p xx ;p)

I contacted the authors via their FB emails but never heard back. Right now it's non-commercial and I was building a commercial product.

How does it compare to Adobe's recent AI audio enhancement?

Not sure. Anyone else wanna chime in on that?

BTW - in my search just now in response to your question, I see Dolby also has an API: https://dolby.io/products/enhance/

Interesting that all of these seem to work extremely well but none are actually available to end users as an unrestricted product.

Somewhat tangential question, have you looked for/found any audio models/tools that can be used for separating out individual voices to separate audio tracks automatically? Perhaps this is already possible with existing tools that I am uninitiated in.

That's called "speaker diarization" and there's quite a bit of work in the field. https://github.com/topics/speaker-diarization

I have no idea what's good or the best, but there's a starting point!

I had speaker diarization and my wife and dog wouldn’t even come near me for weeks

Thank you for the proper terminology, googling these things in plain English just doesn't do it anymore

Not a problem and good luck!

I haven't tested this with multiple voices and it sounds like you want something more specific but it's produced 10/10 results with a couple dozen audio files I've thrown at it, might be of use... https://vocalremover.org/

https://speechbrain.readthedocs.io/en/latest/API/speechbrain...

Izotope RX Pro, which is software for the cleaning and refinement of audio for music and audio post production includes 'Multiple Speaker Detection' which analyzes different voices in a recording and allows you to process them independently.

https://www.izotope.com/en/products/rx.html

I can't speak to it's effectiveness because I don't have any need for it, and also RX 10 Advanced is commercial software and pretty expensive for a casual user, but the feature seems to be on the horizon for other apps.

meta has demucs which is the best I've used so far

Does Ultimate Vocal Remover 5 fit the bill?

Despite the name, it can also do audio separation.

Have you tested https://hushaudioapp.com ? Its quality is amazing.

AFAIU their point was they wanted a reusable library, and specifically NOT a closed-source paid-only solution.

Have you tested the FB one?

Meta does stellar work in AI. I’m quite certain the recent DMCAs were a case of the left hand not knowing what the right hand is doing. This model is open source in every sense of the word, and they’ve now released two impactful models in the span of a month.

This tweet chain is a pretty great overview: https://twitter.com/drjimfan/status/1643647849824161792?s=61...

Are the weights open source?

The weights are not "source".

That is likely true in the most pedantic sense, but in practice, if I create an algorithm that works by using a series of matrix transformations against a set of carefully chosen (read: "trained") matrices and I open-source only the matrix manipulation code but not the specially chosen matrices, I think there's a fair argument to say that I haven't open-sourced the entire algorithm.

In the phrase "the model is open-source in every sense of the word", that, IMO, must include the weights.

I'd be curious to know if you think open source companies should share all of their database records too.

I think of two broad categories of database records: transactional data (data created while running the system) and domain data (data created during development and shipped to production as part of the release process).

The former type of data I wouldn't expect to ever be open-sourced. The latter type might or might not be, depending on the intent of open-sourcing the related system.

If I created a human language translation system that used a SQL database to store the dictionary (domain) data and claimed the system was open-source without shipping the domain data, I think people would rightly say that the system was not fully opened.

The actual source code is not important. The source code can be printed on a single A4 page, the valuable final product is the weights you get after running the code for fifty million dollars of compute time.

If they start claiming they're "open weights" or "open final product" I'll be up in arms.

They are claiming that. They are also delivering on that claim: https://github.com/facebookresearch/segment-anything#model-c...

If I autogenerated a huge amount of C based on the weights, that added/multipled variables the same way the existing code+weights does, then would it be “source”?

The weights seem to be under the same license, just distributed separately because it doesn’t make sense for the giant binary artifacts of training to be part of the source repository.

Yes

Where? From the article:

>Currently, the code (without the weights) is available on GitHub

In the README on said github

https://github.com/facebookresearch/segment-anything#model-c...

The weights are linked from the readme; avoiding large binary resources in GitHub repositories is fairly normal.

Right here: https://github.com/facebookresearch/segment-anything#model-c...

For anyone curious about file sizes for the PyTorch models:

* default/sam_vit_h_4b8939.pth: 2.4GB

* sam_vit_l_0b3195.pth: 1.2GB

* sam_vit_b_01ec64.pth: 358MB

> left hand not knowing what the right hand is doing.

No it was trying to stop abuse. Unlike OpenAI, facebook can't get away with releasing a thing that is more than capable of making up libellous, racist or any other type of illegal or PR decimating thing.

The point of the really restrictive release was to at least try to limit that kind of issue.

Fortunatly for meta, most people who use the weights aren't the end user with a large twitter following, so the risk is (now) low.

The Economist had a recent overview of the AI competitive landscape that highlights Meta's outsize role thanks to their embrace of openness. https://www.economist.com/business/2023/03/26/big-tech-and-t...

They're well on their way to yahoo'ing themselves - valuable and significant software contributions for a platform nobody is using.

I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.

It's funny, back in the 90s I disliked Microsoft and thought they were inherently culturally opposed to open source. But big corporations really just follow their business incentives when it comes to open source.

It's not like Meta "loves to be open". But they would hate a world where all the powerful AI development happened on one of the big clouds. Imagine if every researchers used some closed-source API from Google or Azure or AWS. Meta would try to hire some AI people, and they'd be like, ugh if we take the job we have to use this weird Facebook-specific thing. (<cough>Flow</cough>)

So, supporting PyTorch and providing open source models is just a good business strategy for Meta. I'm glad to see it.

> I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.

My usual reaction to anything Facebook releases is "yawn, they released yet another model that is practically useless because it's released under a non-commercial license", but I'm pleasantly surprised that this one seems to be actually liberally licensed! Hopefully this continues into the future.

I’m glad Microsoft is open sourcing a lot of things.

I have never, ever seen a convincing business case for their doing so, however.

Not your opinion. A compelling business case.

The business case for open sourcing dev tooling is that more developers will use them and build on them and that brings more people to your platforms.

Sure. Any proof at all for that? I mean it's the obvious answer, but generally in business people who don't pay for things tend not to pay for things. In fact, in the enterprise having a big budget is the way you keep your corporate power. Free stuff isn't a way to keep your budget high.

Again, I would like to agree with your thinking. I just don't think the real world backs it up.

It's pretty simple, they make a lot of money off Azure, and open source software is a complement to Azure. See the classic:

https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

In particular, open source software prevents people from getting locked into any of the AWS, Linux, iOS, or Android ecosystems. All of those are popular software development ecosystems that Microsoft does not own or influence very much.

Not convincing or even relevant IMHO. Azure infra is largely closed source and Microsoft is making a ton of money off it.

I'm extremely sorry but I might have submitted a mismatch title and URL. I stumbled on about 5-6 outlets via Twitter.

Here is the announcement from Meta (erstwhile Facebook)

https://ai.facebook.com/blog/segment-anything-foundation-mod...

I've been wanting to build an app that will transform a photo into a template for stained glass. Interactive, to choose the "granule" size, merge color groups into one single piece, change border thickness, etc. Something like this would really help there and make it almost a piece of cake.

Yeah it basically does the first part that with a few lines of code! I've been playing with it using it to segment images and just randomly shake each segment in a short gif. Just to kind of mess around with it until I've found a real use for it or more time to implement some of the more interesting use cases I've got. I'm super pumped this was released!

Perhaps it's not as sophisticated as Meta, but doesn't iOS16 do this?

I recall my partner discovering it by accident whilst messing about with some photos. The people in the foreground were lifted off the background.

Yes, but it is closed to Apple's app only. Android has something something similar, but they haven't released to the public.

Meta is releasing it to everyone, so you can use it on your own apps/services. (and it is a great implementation/state of the art).

Thanks for confirming (wasn't sure if I'd misremembered it somehow!!)

And totally with you that the Meta release is great for getting this kind of thing more widely adopted.

It’s similar but the iOS just segment the main object in an image.

Here's a great summary:

https://twitter.com/DrJimFan/status/1643647849824161792

these are toy examples

If course they are. I haven't looked into this model in particular, but unless I'm way off base, these general purpose pretrained things are completely unsuitable for any specialized (say medical or whatever) application. This seems just like a better version of some of the previous semantic segmentation models they trained on coco.

It's probably a good starting point for fine tuning for custom application though, that's the main use of these big models.

Any idea how/when this will make it into Photoshop?

According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.

But I'm incredibly curious if they'll license this to Adobe, or if Adobe's already got its own team building the same thing that'll be just as good, or what.

It's not super-clear to me why Meta as opposed to Adobe has come out with this first, except maybe to be able to apply filters to just segments of photos in Insta?

> According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.

The article seems to be misleading, since I can just go to the github and download the weights right now: https://github.com/facebookresearch/segment-anything#model-c...

However, for sake of fairness, competition, and reproducible science, I hope that if Adobe makes this a part of their commercial offering, they train their own so we can have two variants of the same model available to us.

Photoshop already has the Object Selection tool: https://helpx.adobe.com/photoshop/using/tool-techniques/obje...

I tried Meta's photo segmentation tool and I didn't see anything magical in it.

I haven't tried Meta's yet but Photoshop's tool though decent struggles with complex scenes

Very meta (pun intended) but is there a website that allows you to say "I want to find an AI engine / application that does X", where X might be any iteration of terms:

"magic wand tool that cuts an image from the background"

"like photoshop where you cut an object from the background"

"photo segmentation tool"

Google would be the de facto here, but its kind of worthless as its gamed by SEO.

Are there any open source alternatives for this? Or still too early?

give it 36 hours and we'll have segMenPoTader.cpp

The model is open source

https://github.com/facebookresearch/segment-anything/

SAM itself _is_ open source: https://github.com/facebookresearch/segment-anything

ASL 2 license.

I wrote a napari-plugin for it, https://github.com/JoOkuma/napari-segment-anything

You just need to have napari installed. I only tested on Linux, so it might not work on Windows/OSX.

This is the open source alternative.

Geez what would I do without another photo segmentation program?

Might retire the 20 year old bash script I use with its advanced Ai if pixelA != pixelB loop.

Unrelated, but can we please get web browsers to enable users to totally turn off all auto play video and video encoding?

I hate websites like this where you deal with a minefield of video areas that might start playing audio at the smallest tap and drain your bandwidth for content you have zero desire to watch.

Many browsers have settings built in for this, and those that don't natively definitely have browser extensions for doing this.

I'm not sure chrome mobile has either.

Yep, Google removed the setting to disable it in version 80 for Android.

I guess my ad blocker or some configuration protected me from it. I didn't encounter a single auto-play video or audio.

Im autistic and have the same Problem. I use chrome extensions which work fine

Firefox can block audio/video autoplay. You can set a default and have per-site exemptions: https://support.mozilla.org/en-US/kb/block-autoplay

My firefox seems to not play it by default.

Or news sites that play some annoying caster talking about it while you're trying to read and it follows you down the page. I'm here for an article not someone reading it to me.

Reply on news.ycombinator.com