Hacker News Reader

Comment by elihu | Hacker News Reader

One idea that's been around for awhile is to identify files by their hash. That has pros and cons. The good side of that is that the file is immutable; you can't accidentally link to something else unless someone can manufacture a hash collision somehow. The down side is that if the file is corrected in some way, you don't get the fixes.

In a lot of the peer-to-peer distributed hash table designs, all you need to retrieve a file is its hash.

https://en.wikipedia.org/wiki/Content_addressable_network

View on HN

Problem is, you have to download the entire video to check the hash. That's not how video embedding works; the client browser is just handed some link, and it obtains pieces of the video, rendering it instantly.

Basically, little segments of the video have to have a signature which is continuously validated. Or something like that.

That's not really a problem. You don't hash the entire video, but do something resembling a merkle tree. i.e. look at a torrent, they're identified by a hash but you can download and verify a random chunk

Right, merkle tree! OK, so the embedding site only stores a single hash: the root one. This hashes the the remaining hashes. The first thing we fetch from the video is those hashes and if their hash doesn't match, we flag/ignore the video and refuse to play.

Multiple levels of the tree can be stored throughout the video file. The first level after the root can be for major sections, like 5 minute segments. The next levels are then at the start of each 5 minute segment, giving hashes for one second chunks.

If the root hash checks out, we get the 5-min hashes. If they check out, we get the hashes for the first 5-min block, and if those hashes check out, we start to play the video, validating every second of it against a one second hash from the 5 min block. Then we get the next 5-min hash block and so on.

Kind of thing.

In actual HLS streaming videos are served in 2-10 second segments, a size small enough that you realistically can hash each segment and verify each segment. Youd have to implement it as an extension of the HLS protocol, so probably as m3u8 with additional fields, maybe as comments.

I don't have expertise in video codecs or file formats, but couldn't you hash the first N bytes of a stream? Stream those N bytes to the client and if it matches start the video, else stop the download and not start the video.

This has a number of problems.

The most egregious is if I'm an attacker and I have the file you request I can hash the appropriate portion you'd use to verify it but fill the rest with junk or exploits. You'd receive the file, it would emit the correct hash, yet be not what you were expecting.

For video especially what you receive isn't necessarily predictable by the client. With HLS or MPEG DASH streaming the video you receive could be one of a number of different encoding e.g. lower or higher bitrates to deal with changing network conditions. The actual m3u8/mpd file you might receive could change arbitrarily as the video provider adds or drops different encodings. The hash of such a file today isn't guaranteed to match the hash tomorrow for entirely banal non-malicious reasons.

Fun fact: the UUHash algorithm used by the FastTrack network (Kazaa, Morpheus, etc) only hashed the first bit of a file. Hashing a large file took forever on hardware of the day. Even hashing small files was non-trivial. The RIAA through various fronts would insert spoofed files where the first portion of the file was legitimate but the content of the file would be junk or annoying sounds. The files would be named like any other MP3 someone was searching for and even have seemingly good IDv3 tags.

> Fun fact: the UUHash algorithm used by the FastTrack network (Kazaa, Morpheus, etc) only hashed the first bit of a file.

The first 300KiB plus a series of 300KiB chunks at exponentially-increasing offsets, per Wikipedia. But still a small fraction of thw file.

Well explained. Thank you!

If you only hash/check the first N bytes of the video stream, the remainder of the video could be anything.

Merkle trees basically accomplish this with separate hashes over every N bytes, so that the content can be verified continuously as it's downloaded.

They will keep the first few seconds or minutes of the original video, bit-exact, and then switch to porn. The player needs to validate every section.

Presumably for SHA256 you only need to hash ~256 bits; what's anyone gonna do, try all possible combinations to find a collision?

256 bits is only 32 bytes, and most file formats have standard stuff right at the start. Collisions would be very common.

That's already how it works in IPFS or Freenet. The hash is not for the data itself, but for a metadata document that contains a list of hashes to the data broken up into smaller blocks.

Assuming there is no malicious intent behind the embeds you could have the hash in the header of the video.

I think that in the context of the topic of this submission "replacement by porn" is being regarded as malicious intent.

Sounds like something subresource integrity[1] could be expanded to include.

[1]: https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

This is were a signature would come in handy, if you want the resource to be updated. Just keep track that it was signed by the same private key.

Reply on news.ycombinator.com