Hacker News Reader

Comment by Delumine | Hacker News Reader

This is insane. Even though there are open-source models, I think this is too dangerous to release to the public. If someone would've uploaded that Tokyo video to youtube, and told me it was a drone.. I would've believed them.

All "proof" we have can be contested or fabricated.

View on HN

"Proof" for thousands of years was whatever was written down, and that was even easier to forge.

There was a brief time (maybe 100 years at the most) where photos and videos were practically proof of something happening; that is coming to an end now, but that's just a regression to the mean, not new territory.

Hmmm. Actually I think I finally figured out why I dislike this argument, so thank you.

The important number here isn't the total years something has been true, when talking about something with sociocultural momentum, like the expectation that a recording/video is truthful.

Instead, the important number seems to me to be the total number of lived human years where the thing has been true. In the case of reliable recordings, the last hundred years with billions of humans has a lot more cultural weight than the thousands of preceding years by virtue of there having been far more human years lived with than without the expectation.

That's a false metric. With exponential progress, we have to adjust equally rapidly. It's quite obvious that photos and videos would last far shorter than written medium as proof of something.

Photos have never been a fundamental proof if the stakes are high or you have an idling censorship institution. Soviets (and maybe others, I just happen to know only about them ) successfully edited photos and then mass-reproduced them.

just some google link about the issue: https://rarehistoricalphotos.com/stalin-photo-manipulation-1...

This changes nothing about "proof" (i.e. "evidence", here). Authenticity is determined by trust in the source institution(s), independent verification, chains of evidence, etc. Belief is about people, not technology. Always was, always will be. Fraud is older than Photoshop, than the first impersonation, than perhaps civilization. The sky is not falling here. Always remember: fidelity and belief aren't synonyms.

Scale matters. This will allow unprecedented scale of producing fabricated video. You're right about evidence, but it doesn't need to hold up in court to do a lot of damage.

No, it doesn't. You cannot scale your way into posting from the official New York Times account, or needing valid government ID to comment, or whatever else contextually suggests content legitimacy. Abusing scale is an ancient exploit, with myriad antidotes. Ditto for producing realistic fakes. Baddies combining the two isn't new, or cause for panic. We'll be fine.

Your entire argument that scale doesn't matter rests on the notion that legitimacy needs to be signalled at all to fool people. It doesn't. It just needs to appeal to people's biases, create social chaos through word of mouth. Also, all you need to get posted on the NY times "account" is to fool some journalists. Scale can help there too by creating so much misinformation it becomes hard to find real information.

Scale definitely matters when that's what you're doing. In fact I challenge you to find any physical or social phenomenon where scale doesn't matter.

If read aloud, no one could guess if your comment came from 2024 or 2017. There is zero barrier between you and using trusted sources, or endlessly consuming whatever fantasy bullshit supports your biases. That has not, and will not, change.

Look, you can repeat all you want that fraud has existed before, but that's not an argument.

> All "proof" we have can be contested or fabricated.

This has been the case for a while now already, it's better that we just rip off the bandaid and everyone should become a skeptic. Standards for evidence will need to rise.

If you rip off the bandaid too soon, there will be blood.

That's interesting. It made me think of a potential feature for upcoming cameras that essentially cryptographically sign their videos. If this became a real issue in the future, I could see Apple introducing it in a new model. "Now you can show you really did take that trip to Paris. When you send a message to a friend that contains a video that you shot on iPhone, they will see it in a gold bubble."

Weird hallucination artifacts are still giving it all away. Look closely at the train and viaduct rendering, and you can't unsee windows morphing into each other.

The key word there is "someone". The only way forward is to care a lot more about our sources. Trust is about to become really valuable.

We give too much credit to ordinary people. All these bleeding-edge advancements in AI, code, databases, and technology are things a user on HNews would be aware of. However, most peers in regular jobs, parents, children, et al., would be susceptible to being fooled on social media. They're not going to say... "hmm, let me fact-check and see if the sources are correct and that this wasn't created by AI."

They'll simply see an inflammatory tweet from their leader on Twitter.

They're not going to fact check, they're simply going to think "huh, could be AI" and that will change the way we absorb and process information. It already has. And when we really need to know something and can't afford to be wrong, we'll seek out high trust sources. Just like we do now, but more so.

And of course some large cross section of people will continue to be duped idiots.

Most people don't even know what AI is. I've had to educate my parents that the technology to not only clone my voice, but my face.. is in existence. Pair that with number spoofing, and you have a recipe for disaster to scam people.

This is what lots of folks said about image generation. Which is now in many ways “solved”. And society has easily adapted to it. The same will happen with video generation.

The reality is that people are a lot more resourceful / smarter than a lot of us think. And the ones who aren’t have been fooled long before this tech came around.

In what ways has image generation been solved? Prompt blocking is about the only real effort I can think of, which will mean nothing once open source models reach the same fidelity.

I guess you can't read Japanese.

maybe for now, only a matter of time before stuff like this is fixed

the world is getting increasingly surveilled as well, I guess the presumption is that eventually you'll just be able to cross reference a 'verified' recording of the scene against whatever media exists.

"We ran the vid against the nationally-ran Japanese scanners, turns out that there are no streets that look like this, nor individuals."

in other words I think that the sudden leap of usable AI into real life is going to cause another similar leap towards non-human verification of assets and media.

And I guess you haven't actually been to Tokyo, the number of details which are subtly wrong is actually very high, and it isn't limited to text, heck detecting those flaws isn't even limited by knowledge of Japan:

- Uncanny texture and shape for the manhole cover

- Weirdly protruding yellow line in the middle of the road, where it doesn't make sense - Weird double side-curb on the right, which can't really be called steps.

- Very strange gait for the "protagonist", with the occasional leg swap.

- Not quite sensical geometry for the crosswalks, some of them leading nowhere (into the wet road, but not continuing further)

- Weird glowy inside behind the columns on the right.

- What was previously a crosswalk, becoming wet "streaks" on the road.

- No good reason for crosswalks being the thing visible in the reflection of the sunglasses.

- Absurd crosswalk orientation at the end. (90 degrees off)

- Massive difference in lighting between the beginning of the clip and the end, suggesting an impossible change of day.

Nothing suggests to me that these are easy artifacts to remove, given how the technology is described as "denoising" changes between frames.

This is probably disruptive to some forms of video production, but the high-end stuff I suspect will still use filming mostly ground in truth, this could highly impact how VFX and post-production is done, maybe.

With everything we've seen in the last couple years, do you sincerely believe that all of those points won't be solved pretty soon? There are many intermediary models that can be used to remove these kind of artefacts. Human motion can be identified and run through a pose/control-net filter, for example. If these generations are effectively one-shot without subsequent domain-specific adjustments, then we should expect for every single one of your identified flaws to be remedied pretty soon.

all the news you see has zero proof unless you see it, you just have to have a sense if it's real based on a concensus or trust worthness of a reporter/outlet.

The UA war is real, most likley, but i havent' seen it with my own eyes, nor did most people, but maybe they have relatives/friends saying it, and they are not likely to lie. Stuff like that.

Reply on news.ycombinator.com