Hacker News Reader

Comment by paulsutter | Hacker News Reader

It’s really easy for them to block this method: return a random video for a certain percent of non existing identifiers. Throw in a bit of random for good measure

This is the risk of explaining the method

View on HN

My guess is that would be very, very difficult without breaking a ton of other invariant assumptions throughout the system, like:

Video IDs are immutable

A video can only be represented by a single unique video ID

etc.

You could keep that assumption and just serve a random video if some external IP dials a non-existent ID.

Presumably, no one would do that except researchers trying to count videos (or randomly find hidden ones?).

You can break the assumption of unicity (if an unassigned ID is later assigned) if you do that internally, although not sure that’d be common but it’s not an assumption that has to be strict for non-attributed ones, and you never use the fake ID.

This is easy to detect, though, at least for public videos. Click through to the channel on a different IP and find the video link, or search for the video's title and description, or find a canonical link to the video by any other means. If the IDs don't match, it's been faked.

Maybe use a hash for consistency?

You misunderstand that this technique was executed by using YouTube _search_ to find videos, not by querying the exact URL. They can doctor search results however they like.

> return a random video for a certain percent of non existing identifiers.

if you got a video from a randomly generated ID, you can immediately query for it again, and see if the video was the same as before.

If it's not the same, you disgard the result and assume the ID generated was actually non-existent.

If it's the same, then you know it's a real ID.

As long as youtube vidoe URLs are immutable, this method will stop the blocks you described.

Unless they cache the false information…

but now you have to store state for some arbitrary url, share it across many different clusters/regions, record which IP has seen it etc...

a lot of work, for very little gain.

You could use a hashing function that guarantees the same result every time.

Though if they didn’t say they were doing that we wouldn’t know the method was invalid. Further, that other video would have its own existing uid, so in theory we could know if they’d duplicated it to thwart these measures.

I can’t imagine trying to debug production errors with a system like this.

Are the video IDs sequential in the available domain? or just all over the place? Is there anything in common with all known live video IDs that could make it easier to scan the quintillion possibilities?

They could use a random redirect function. Throw the user a video URL which is random, but then redirects to the video that you want. So one could never count the space?

I think this would be easily detectable anyway as whatever IP’s they are using would be viewing videos that basically no one ever watches.

The paper says they made 1.8e10 attempts to produce 1e4 videos. TFA says they now have 2.5e4 videos, so they’ve made >4e10 attempts so far. No way in hell you can scrape YouTube like that without buying access to a big proxy IP pool (e.g. Luminati).

Or don't return any search results for searches for random hashes

Reply on news.ycombinator.com