Like it would be nice if all photo and video generated by the big players would have some kind of standardized identifier on them - but now you're left with the bajillion other "grey market" models that won't give a damn about that.
Like it would be nice if all photo and video generated by the big players would have some kind of standardized identifier on them - but now you're left with the bajillion other "grey market" models that won't give a damn about that.
> Were politicians 20 years ago as overreative they'd have demanded Photoshop leave a trace on anything it edited.
The arguments put forward by people generally I don't find compelling -- for example, in this thread around protecting against counterfeit.
The "force" applied to address these concerns is totally out of proportion. Whenever these discussions happen, I feel like they descend into a general viewpoint, "if we could technically solve any possible crime, we should do everything in our power to solve it."
I'm against this viewpoint, and acknowledge that that means _some crime_ occurs. That's acceptable to me. I don't feel that society is correctly structured to "treat" crime appropriately, and technology has outpaced our ability to holistically address it.
Generally, I don't see (speaking for the US) the highest incarceration rate in the world to be a good thing, or being generally effective, and I don't believe that increasing that number will change outcomes.
I think that by now it should be crystal clear to everyone that it matters a lot the sheer scale a new technology permits for $nefarious_intent.
Knives (under a certain size) are not regulated. Guns are regulated in most countries. Atomic bombs are definitely regulated. They can all kill people if used badly, though.
When a photo was faked/composed with old tech, it was relatively easy to spot. With photoshop, it became more complicated to spot it but at the same time it wasn't easy to mass-produce altered images. Large models are changing the rules here as well.
I don’t think this is a good comparison: knives are easy to produce, guns a bit harder, atomic bombs definitely harder. You should find something that is as easy to produce as a knife, but regulated.
Or, if you see the altered photo as the "product", then the "product" of the knife/gun/bomb is the damage it creates to a human body.
The DEA and ATF have entered the chat
The story of human history is newer generations freaking about progress and novel changes that have never been seen before. And later generations being perfectly okay with it and adapting to a new style of life.
The issue is that some people believe shit someone tells them and they deny any facts. This has been always a problem. I am all in for labeling content as AI generated. But it wont help with people trying to be malicious or who choose to be dumb. Forcing to watermark every picture made neither, it will turn into a massive problem, its a solid pillar towards full scale surveillance. Just alone the fact that analog cams become by default less trustworthy then any digital device with watermarking is terrible. Even worse, phones will eventually have AI upscaling and similar by default, you can't even make an accurate picture without anything being tagged AI. The information is eventually worthless.
We could use the opportunity to deploy robust systems of verification and validation to all digital works. One that allows for proving authenticity while respecting privacy if desired. For example… it’s insane in the US we revolve around a paper social security number that we know damn well isn’t unique. Or that it’s a massive pain in the ass for most people to even check the hash of a download.
Guess which we’ll do!
For example, I'd rather read a tweet from Paul Graham's account than read a screenshot of a Paul Graham tweet that a stranger emailed to me. In the latter case, the source makes me suspicious, so then I go to a more trusted source, e.g. a reputable news organization, or PG's actually X account.
https://petapixel.com/2024/01/02/cameras-content-authenticit...
seems like a better way
But people with actual nefarious intent will easily be able to remove these watermarks, however they're implemented. This is copy protection and key escrow all over again - it hurts honest people and doesn't even slow down bad people.
IOW - People aren't turning off their brains about "for the children" - they just want it anyway and don't think any further than that.
https://www.nbcnews.com/tech/tech-news/ai-generated-evidence...
> “My wife and I have been together for over 30 years, and she has my voice everywhere,” Schlegel said. “She could easily clone my voice on free or inexpensive software to create a threatening message that sounds like it’s from me and walk into any courthouse around the country with that recording.”
> “The judge will sign that restraining order. They will sign every single time,” said Schlegel, referring to the hypothetical recording. “So you lose your cat, dog, guns, house, you lose everything.”
At the moment, the only alternative is courts simply never accept photo/video/audio as evidence. I know if I were a juror I wouldn't.
At the same time, yeah, watermarks won't work. Sure, Google can add a watermark/fingerprint that is impossible to remove, but there will be tools that won't put such watermarks/fingerprints.
https://en.wikipedia.org/wiki/Printer_tracking_dots
Yes, can we not jump on the surveillance/tracking/censorship bandwagon please?
I bet it will be called "Real Photos" or something like that, and the pictures will be signed by the camera hardware. Then iMessage will put a special border around it or something, so that when people share the photos with other Apple users they can prove that it was a real photo taken with their phone's camera.
There used to be a joke about people who did slideshows (on an actual slide projector) of their vacation photos at parties.
How "real" are iPhone photos? They're also computationally generated, not just the light that came through the lens.
Even without any other post-processing, iPhones generate gibberish text when attempting to sharpen blurry images, they delete actual textures and replace them with smooth, smeared surfaces that look like a watercolor or oil paintings, and combine data from multiple frames to give dogs five legs.
You're right that there will existed generated content without these watermarks, but you can bet that all the commercial providers burning $$$$ on state of the art models will gradually coalesce around some means of widespread by-default/non-optional watermarking for content they let the public generate so that they can all avoid drowning in their own filth.
I don't see how it would defeat the cat and mouse game.
For example, it's trivial to post an advertisement without disclosure. Yet it's illegal, so large players mostly comply and harm is less likely on the whole.
It still won't prevent it, but it would prevent large players from doing it.
Plus, any service good at reverse-image search (like Google) can basically apply that to determine whether they generated it.
There will always be a way to defeat anything, but I don't see why this won't work for like 90% of cases.
Always has been so far. You add noise until the signal gets swamped. In order to remain imperceptible it's a tiny signal, so it's easy to swamp.
It may be easier if you have an oracle on your end to say "yes, this image has/does not have the watermark," which could be the case for some proposed implementations of an AI watermark. (Often the use-case for digital watermarks assumes that the watermarker keeps the evaluation tool secret - this lets them find, e.g, people who leak early screenings of movies.)
No, but model training technology is out in the open, so it will continue to be possible to train models and build model toolchains that just don't incorporate watermarking at all, which is what any motivated actor seeking to mislead will do; the only thing watermarking will do is train people to accept its absence as a sign of reliability, increasing the effectiveness of fakes by motivated bad actors.
have some kind of standardized identifier on them
Take this a step further and it'll be a personal identifying watermark (only the company can decode). Home printers already do this to some degree.All of this is trivially easy to circumvent ceremony.
Google is doing this to deflect litigation and to preserve their brand in the face of negative press.
They'll do this (1) as long as they're the market leader, (2) as long as there aren't dozens of other similar products - especially ones available as open source, (3) as long as the public is still freaked out / new to the idea anyone can make images and video of whatever, and (4) as long as the signing compute doesn't eat into the bottom line once everyone in the world has uniform access to the tech.
The idea here is that {law enforcement, lawyers, journalists} find a deep fake {illegal, porn, libelous, controversial} image and goes to Google to ask who made it. That only works for so long, if at all. Once everyone can do this and the lookup hit rates (or even inquiries) are < 0.01%, it'll go away.
It's really so you can tell journalists "we did our very best" so that they shut up and stop writing bad articles about "Google causing harm" and "Google enabling the bad guys".
We're just in the awkward phase where everyone is freaking out that you can make images of Trump wearing a bikini, Tim Cook saying he hates Apple and loves Samsung, or the South Park kids deep faking each other into silly circumstances. In ten years, this will be normal for everyone.
Writing the sentence "Dr. Phil eats a bagel" is no different than writing the prompt "Dr. Phil eats a bagel". The former has been easy to do for centuries and required the brain to do some work to visualize. Now we have tools that previsualize and get those ideas as pixels into the brain a little faster than ASCII/UTF-8 graphemes. At the end of the day, it's the same thing.
And you'll recall that various forms of written text - and indeed, speech itself - have been illegal in various times, places, and jurisdictions throughout history. You didn't insult Caesar, you didn't blaspheme the medieval church, and you don't libel in America today.
How can they distinguish from real people exploited to AI models autogenerating everything?
I mean right now this is possible, largely because a lot of the AI videos have shortcomings. But imagine in 5 years from now on ...
The people who care don't consume content which even just plausibly looks like real people exploited. They wouldn't consume the content even if you pinky promised that the exploited looking people are not real people. Even if you digitally signed that promise.
The people who don't care don't care.
Watermarking by compliant models doesn't help this much because (1) models without watermarking exist and can continue to be developed (especially if absence of a watermark is treated as a sign of authenticity), so you cannot rely on AI fakery being watermarked, and (2) AI models can be used for video-to-video generation without changing much of the source, so you can't rely on something accurately watermarked as "AI-generated" not being based in actual exploitation.
Now, if the watermarking includes provenance information, and you require certain types of content to be watermarked not just as AI using a known watermarking system, but by a registered AI provider with regulated input data safety guardrails and/or retention requirements, and be traceable to a registered user, and...
Well, then it does something when it is present, largely by creating a new content gatekeepiing cartel.
So, you exploit real people, but run your images through a realtime AI video transformation model doing either a close-to-noop transformation or something like changing the background so that it can't be used to identify the actual location if people do figure out you are exploiting real people, and then you have your real exploitation watermarked as AI fakery.
I don't think this is solving a problem, unless you mean a problem for the would-be exploiter.
Exactly. When the barrier to entry for training a okay-ish AI model (not SOTA, obviously) is only a few thousand compute hours on H100s, you couldn't possibly hope to police the training of 100% of new models. Not to mention that lots of existing models are already out there are fully open-source. There will always be AI models that don't adhere to watermark regulations, especially if they were created a country that doesn't enforce your regulations.
You can't hope to solve the problem of non-watermarked AI completely. And by solving it partially by mandating that the big AI labs add a unified watermark, you condition people to be even more susceptible to AI images because "if it was AI, it would have a watermark". It's truly a no-win situation.
If social media platforms are required by law to categorize content as AI generated, this means they need to check with the public "AI generation" providers. And since there is no agreed upon (public) standard for imperceptible watermarks hashing that means the content (image, video, audio) in its entirety needs to be uploaded to the various providers to check if it's AI generated.
Yes, it sounds crazy, but that's the plan; imagine every image you post on Facebook/X/Reddit/Whatsapp/whatever gets uploaded to Google / Microsoft / OpenAI / UnnamedGovernmentEntity / etc. to "check if it's AI". That's what the current law in Korea and the upcoming laws in California and EU (for August 2026) require :(
We will always have local models. Eventually the Chinese will release a Nano Banana equivalent as open source.
If watermarking becomes a legal mandate, it will inevitably include a prohibition on distributing (and using and maybe even possessing, but the distribution ban is the thing that will have the most impact, since it is the part that is most policable, and most people aren't going to be training their own models, except, of course, the most motivated bad actors) open models that do not include watermarking as a baked-in model feature. So, for most users, it'll be much less accessible (and, at the same time, it won't solve the problem.)
As long as someone somewhere is publishing models that don't watermark output, there's basically nothing that can stop those models from being used.
https://generative-ai.review/2025/09/september-2025-image-ge... (non-pro Nano Banana)
Hell, it might even be possible for some arbitrary photographs to come up with an AI prompt that produces them or something similar enough to be indistinguishable to the human eye, opening up the possibility of "proving" something is fake even when it was actually real.
What you want just can't work, not even from a theoretical or practical standpoint, let alone the other concerns mentioned in this thread.
Maybe zero knowledge proofs could provide anonymity, or a simple solution is to ship the same keys in every camera model, or let them use anonymous sim-style cards with N-month certificate validity. Not everyone needs to prove the veracity of their photos, but make it cheap enough and most people probably will by default.
Unless the watermark randomly replaces objects in the scene with bananas, these images/videos will still spread like wildfire on platforms like TikTok, where the average netizen's idea of due diligence is checking for a six‑fingered hand... at best.
And if it can be seen like that, it should be removeable too. There are more examples in that thread.
Some supposition: A Fourier amplitude image should show that pattern as peaks at a certain angle/radius location. The exact location may be part of the identification scheme. Running peak finding on the Fourier image and then zeroing out the frequencies in the peak should remove the pattern. Modeling the shape of the peak would allow mimicking the application of a legit SynthID signature.
If anyone tries/tried this already, I'd love to see the results.
Image verification has never been easy. People have been airbrushed out of and pasted into photos for over a century; AI just makes it easier and more accessible. Expecting a “click to verify” workflow is unreasonable as it has ever been; only media literacy and a bit of legwork can accomplish this task.
Photo-of-a-screen: https://gemini.google.com/share/ab587bdcd03e
It reported 25-50% for the image without having been through that analog hole: https://gemini.google.com/share/022e486fd6bf