Hacker News Reader

Comment by iforgotpassword | Hacker News Reader

I'm probably being captain obvious here, but if this is what's being released for free, I wonder how much better a polished commercial version does, and when we reach the point where we can't trust anything we see anymore. It doesn't even have to be super perfect, even reaching the point where it takes experts about two weeks to determine if something's real or not might already be long enough to do great damage.

From a technical standpoint I think this is very impressive, and I'm also interested in creative/artsy use of this. Their "replace trees by houses" example is pretty dull but gives a good glimpse at what can be done.

View on HN

AI can also be used to identify these fake videos, and they are probably better than human identification. There will be a rise of AI forensics.

The type of neural network they use (GAN) works by having two networks battling each other, one tries to generate fake, and another (discriminator) tries to identify fake, it's a constant arms race. As the generator gets better, the discriminator also gets better. Which means, if the fake video is this good, there must be a discriminator network that identifies fakes just as good.

We did a similar project using GAN, generating images from a text description. You can see the progression of generator and discriminator battling each other, and both get better with time.

https://github.com/yonkshi/text2imageNet

This is how their model was trained, but I think what you've said may not quite be the case.

Because the discriminator (D) and generator (G) usually compete in a minimax game, the equilibrium probability of D correctly classifying an image as fake tends to 1/2 (ignoring distributional factors). If the competing networks have enough capacity and can be stably trained, then in theory they will reach equilibrium as the data distribution from G converges to the actual data distribution. If this is the case, then the discriminator correctly identifies fake videos with a probability of 1/2.

They may not reach equilibrium (making D > 0.5), but it's not clear that the discriminator itself is a panacea for identifying fake videos/images.

The whole GAN thing is useless though if neither of the models have a concept of symmetry or that those pixels do represent actual solid objects in space. Modern neural nets can't represent abstract spatial and semantical concepts or reason, so that videos are full of glaring perspective inconsistencies and large-scale artifacts. So nope, neither AI can help to "identify fake videos", nor do we need AI for that.

The problem is that it's a game of superhuman cat and mouse.

We'll have systems arguing with each other and no way to tell which is correct. If, for example, someone is able to get a copy of the forensic AI model, they can train their decoder/generator to work against it until the results pass as legitimate. With no human ability to argue with the results of the forensics AI, we'll just trust it and pass it off as truth.

If you can clone the jury and conduct a quadrillion private trials, your chance of success in court is going to increase substantially.

Things gonna get creepy.

It could also lead to crimes like Blackmail becoming extinct. It would be hard to hold incriminating recordings of anyone over them if near-perfect audio and video synthesis was common.

Especially for public figures with lots training data available.

That's equally horrifying in a different way.

Also, map images pulled from facebook to the bodies of pornstars. The creepiness and invasion of ... personal image(?) this enables is horrifying.

That's already happened. Search for deep fakes. We already have face substitution in videos which is working surprisingly well sometimes.

There's been r/deepfakes where around 30% of the content was porn with swapped faces. It was banned though.

It was pretty limited though. For split seconds videos seemed real but the seams soon enough showed.

Fake celebrity porn has been on the internet since 1996 at the very least. It's always been crummy; but porn in general requires a thick suspension of disbelief and an intense focus in a partial object of desire (what Lacan calls the objet petit a) that blurs everything else.

Maybe but only after extensive proliferation.

Initially, every Tom, Dick & Harry script kiddie will try to blackmail people whilst the public and justice system is not really aware of the technology possible.

When eventually enough of the public is made aware to not trust photos and videos anymore then these types of blackmail attempts would be less effective. But there will still be less informed gullible targets.

Surely lawyers will know about the tech soon enough to simply show the same video evidence with the judge in place of the defendant -- reasonable doubt right there.

The blackmail angle I expect will work for a while. People falsely accused of sexual crimes can have their lives ruined by the accusation.

The justice system will hopefully catch up quickly.

But as you say the media (real and social), political propaganda, etc will exploit the uninformed masses for all it is worth, so blackmail will still work. For a while.

Yeah... The problem is try explaining deepfakes to your significant other when they are randomly sent what looks like a video of you cheating on them. Sure it’s possible but not likely.

It's all a matter of cultural awareness. Everyone now thinks when seeing an unlikely photo -- "Photoshop?"

This stuff even has the catchy name "deep fake".

DRM will be pushed hard, starting from video/audio acquisition, perhaps assisted by blockchain to keep footage verified at all processing steps.

Don't you think that a blockchain that works for anything other than a rather useless currency should be created before suggesting one for such a use? I see comments all the time about how we should use blockchain for this and that, and yet far simpler uses for blockchain haven't yet worked out.

The sentiment and idea is there, just need the implementation.

A blockchain is actually a reasonable part of the solution here (just not nessesarily creating a new one). What a blockchain really does is decentralized timestamping. Usually you use it to combine it with a bit of cryptography to process flows of money, but in the same blockchain you can just write "I know hash XXXX", and if later you produce a work with that hash you can prove that it already existed at the point your message was written in the blockchain.

That's not all you would need for verification, but it is a big help.

Someone's gonna come up with a simple way to sign pics and videos with public key encryption, after which signed media will be the only stuff that is trusted. Of course people will still make sex tapes and sign them when they are drunk because the signing software will be automatically integrated into the camera app, so blackmail is still a possibility.

But.. if the camera app automatically signs pics/vids, that would mean that a private key is available to the app without any passphrase (or one embedded in the app :O).. So why not just extract they key and sign your fake vid?

Well, for something like this the user's signing key itself would probably be managed by the OS, so not extractable. When the app is done editing, it asks the user via the OS to approve the result and the OS performs the signature. You could even embed a downsample of the original that came signed from the camera hardware.

If the key is managed by the OS but is inaccessible to the user, the concept would seem to be incompatible with free software operating systems. Also, if the camera app can have anything it "makes" approved, the app itself could take a deepfake video (from the web or device storage) and have the OS sign it.

The only way I see it working is if the key is "burnt-in" to in the camera hardware and any applications cannot MitM it.

Why would you assume it would be inaccessible to the user? I'd expect a key management interface at least. I guess I did say "not extractable", but I meant more that random apps don't have access to it directly, but they call an api to do the signing.

> the app itself could take a deepfake video and have the OS sign it

Note how I said "it asks the user via the OS to approve the result". I would expect a modal OS dialog to let the user review and approve the content before being signed and passed back to the app.

Thinking about it, there's actually nothing stopping this from happening on today's hardware using just application sandboxing. Substitute "OS" above with "Signing App" that does the same thing (accepts media signature requests from other apps, and opens dialog to request approval from user with a preview).

Isn’t this something a lookalike and some good make up could already do for ages?

That would require an entire team's worth of skill, preparation and work. This could be done with a few already existing medias, and a single person behind a computer.

Yeah, the Tweet I found this from had a similar sentiment:

https://twitter.com/PiratePartyINT/status/104296466807811686...

"Starting now, we cannot trust video or audio evidence. The ramifications for our legal & political systems will not be known for many years"

"Starting now"? How's about 5 to 10 years ago. This is being released free now, which indicates to me this is now disposable tech and the authors have much, much better in their labs.

Probably not; the GAN technique wasn't even published until 2014, and this is based on other work that's been done even more recently. The field moves fast and is more open than you might expect (because it's still so academic).

Usually it's the other way around with ML researchers - what they show is better than what they have in the lab :)

Not sure it works this way except in extremely specific areas of research (and I'm not even sure about that).

Publicly available research is usually the cutting edge in most areas of knowledge.

If you compare OSS or free software to commercial software generally, I don’t think there are that many massive gaps. It’s mostly polish and small incremental improvements, but the underlying tech is mostly the same. Why would that be different in this case?

Adobe have definitely bested most oss competitors in their space. Although with the amount of man power at their disposal it would be hard to beat.

The vast bulk of Adobe's advantage is in UX, not technical algorithms. Which makes perfect sense because that tends to be the case with most F/OSS software—technically brilliant but with an face only a programmer could love.

Yes, Adobe do have some remarkable algorithms that would be difficult to replicate (e.g. heal brush and content aware fill) but these are a small minority of Adobe's software advantage.

The one that irritates me the most is vector drawing programs: open source programs (and even paid competitors) just can't touch Adobe Illustrator for the sort of work I do. I'm sure at least 50 percent of it is familiarity and muscle memory, but I've desperately tried switching to a few different options like Inkscape or Affinity and left wildly disappointed.

for content aware fill and heal brush both GMIC ( https://patdavid.net/2014/02/getting-around-in-gimp-gmic-inp...) and resynthetizer ( http://www.logarithmic.net/pfh/resynthesizer ) are working quite OK.

Gimp even had content aware fill first, It was based on a SIGGRAPH paper iirc.

Do you use the Astute plugins for AI, or the native pen tool? Affinity feels different, maybe less precise, but the functionality was way better compared with the native Adobe tool. You should try Figmas pen tool, I like it.

My uses are relatively trivial- logo design, SVG generation, basic layout work and PDF tinkering. My main need is fine control of beziers with auto-guides to ensure consistency.

You're right that UX is one of the biggest problems they have. One thing that is also hard to replicate it how well Adobe's software works together. Embedding smart objects and illustrator files in photoshop documents, right clicking a clip in Premiere and sending it to After Affects and back again without rendering an intermediate file etc.

I would be interested in a Lightroom alternative if anyone can recommend one though.

On average I agree they are on par. Sometimes OSS is better (ffmpeg), sometimes commercial stuff (adobe after effects).

In this specific case I think it might be beneficial if you can spend a lot of money on gathering training material and tweaking the network. But then again someone here mentioned the quality 4chans fake porn has reached, so maybe I'm wrong after all.

This might actually be one of those edge cases where commercial is better, but the government's secret version is by far the best.

Like what though? Historically what tech does did governments have that was unknown to commercial domains?

People are motivated by money, governments pay little and there’s no way to get rich working there so it doesn’t make that much sense.

Around WWII Computers, Radar, Nuclear power, crypto, jet aircraft, rockets.

Around 1960 space flight, supersonic flight.

More recently hypersonic flight, and we don’t really know becase people back in WWII had no idea.

True but now all that stuff is pretty much produced in the private sector by government contracts with these big companies. DARPA doesn’t have better scientists than MIT and there are a lot of extremely intelligent scientists who have moral issues about working for the military.

MS Office is still miles ahead of any OSS “alternative“, despite the many glaring bugs.

MS Office is still miles ahead in the benchmark of opening it's own proprietary files.

There is no competitor, proprietary or open, that comes close to Excel. It's been relentlessly, extensively polished for years and years, and keeps gaining new features every year.

And this sticking to the spreadsheet concept, which is very limiting.

---

Contrast for example Tableau -- it's a great idea and generated a lot of enthusiasm for a while, but never quite took off as an office package one needs to have. The normal awkwardness of its first versions is still there; they don't have the deep <whatever> that the Excel team has.

Tableau is great, but it has a much narrower use case: given one or more tables of data, generate graphs for presentation or for exploring the dataset.

In comparison, Excel can do that too (just worse), but it can also solve equations, do your company's bookkeeping, and pretty much every other task that relies mostly on numbers.

I would argue Open/LibreOffice Calc comes fairly close to Excel if you ignore the worse user interface (which is fair in the original assertion that it's "mostly polish and small incremental improvements")

> if you ignore the worse user interface

considering that's a major part of "better" that's big ask!

Yes, but given switching costs and habit formation, why would people care about something that's not strictly Pareto dominant?

The gaps are massive in several electrical/silicon CAD verticals and simulation. Due to IP secrecy, no OSS Verilog/VHDL synthesis alternative exists for Vivado/ISE/Quartus and the Intel/Xilinx line of FPGAs. I don't think any practical alternatives exist for Mentor Graphics' line of silicon design or simulation software, nor have I seen any OS software capable of complex mixed domain simulations like COMSOL or Ansys - many of the pieces exist, but it takes a lot of work to verify that algorithmic physical models actually work together.

I'm sure the point is already crossed. People on 4chan are making fake porn that is nearly indistinguishable - and they are complete amateurs.

Are you sure they're amateurs, not professionals don't it for fun (/pro boner/!?!).

I meant something a little different. Any random dude with a random mid-level PC can download the software and produce amazing results without any special hardware or knowledge. I recommend you to try it out for yourself.

Was there a huge progress made since news about deepfakes broke early this year?

Around March/April this year, I actually did download the TensorFlow toolkit for face transfer that was used by /r/deepfakes people and tested it out (there were samples of photos of politicians included); the results were, at best, worse that I could do in 2 minutes in Gimp. Maybe they could get better if I had an expensive GPU farm at my disposal, but I'm pretty doubtful - given that the news died down pretty quickly, and no reasonable-quality faked pictures or videos were reported ever since.

It's more about experimenting with your training data and other configuration. What I've heard, getting great results takes time - but it's possible. Most of the focus of the community is on porn, so I can imagine not so many journalists are checking the newest results and reporting about them.

Probably Nvidia is not interesting in this type of tools and this is more like "hey, look what you can do with a lot of Nvidia cards, buy us a lot of them and you will do this and much more"

Yeah but it was already the case with CGI. IT's true that it's going to be easier and easier to do fake porn, fake speeches, fake voice recording, fake vides....

We reached that point of not being able to trust any media a while ago. It's just getting easier and more widely distributed.

Honestly, you just need to convince one person to execute a successful social engineering attack.

Reply on news.ycombinator.com