Hacker News Reader

Comment by rococode | Hacker News Reader

Clicked into a couple random videos, looks like all of their video IDs are 11 characters, alphanumeric with cases. So 26+26+10 = 62 choices for each char, 62^11 = 5.2e+19 = 52 quintillion unique IDs (52 million trillions).

So, yeah, sampling would be a mostly futile effort since you're looking to estimate about 8 to 10 decimal digits of precision. Though it's technically still possible since you'd expect about 1 in every 50 million - 5 billion IDs to work (assuming somewhere between a trillion and 10 billion videos).

My statistics knowledge is rusty, but I guess if you could sample, say, 50 billion urls you could actually make a very coarse estimate with a reasonable confidence level. That's a lot but, ignoring rate limits, well within the range of usual web-scale stuff.

View on HN

I tried this for some time, I was looking for unlisted videos.

Just generate a random valid link and then check if it gives a video or not.

I found exactly 0 videos.

They also use "_" and "-" according to Tom Scott.

https://www.youtube.com/watch?v=gocwRvLhDf8

Which would bring it up to a nice 64 choices, making it exactly 6 bits per character.

It's a URL-friendly form of base64.

11 chars encode 66 bits, but actually 2 bits are likely not used and it's simply an int64 encoded to base64.

Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet.

Int64 is my go to PK as well, if I have to, I make it hierarchical to distribute it, but I don't do UUID.

> Given everyone and their grandma is pushing 128-bit UUID for distributed entity PK, it's interesting to see YouTube keep it short and sweet.

The trade-off you make when using short IDs is that you can't generate them at random. With 128-bit Id, you can't realistically have collisions, but with 64-bit ones, because of the birthday paradox, as soon as you have more than 2^32 elements, you're really likely to have collisions.

The reason why we want ids to be purely random is so we don't have to do the work of coordinating distributed id generation. But if you don't mind coordinating, then none of this matters.

Surely if it was a great chore for YouTube to have random-looking int64 ids, they would switch to int128. But they haven't.

I'm a big fan of the "works 99.99999999% of the time" mentality, but if anything happens to your PRNGs, you risk countless collisions to slip up by you in production before you realize what happened. It's good to design your identity system in a way that'd catch that, regardless of how unlikely it seems in the abstract.

The concept of hierarchical ids is undervalued. You can have a machine give "namespaces" to others, and they can generate locally and check for collisions locally in a very basic way.

> but if anything happens to your PRNGs, you risk countless collisions to slip up by you in production before you realize what happened.

UUID generation basically has to use a CSPRNG to avoid collisions (or at least a very large-state insecure PRNG).

Because of the low volume simply using /dev/urandom on each node makes the most sense. If /dev/urandom is broken so is your TLS stack and a host of other security-critical things; at that point worrying about video ID collisions seems silly.

I worry about state corrupting problems, because they tend to linger long after you have a fix.

Youtube video ids used to be just base64 of a 3DES-encrypted mysql's primary key, a sequential 64-bit int - collisions are of zero concern there. By birthday paradox it's about as good as 128-bit UUID generated without using a centralized component like database's row counter, when you have to care about collisions.

However theft of the encryption key is a concern, since you can't rotate it and it just sat there in the code. Nowadays they do something a bit smarter to ensure ex- employees can't enumerate all unlisted videos.

You seem to know about their architecture. What do they do now?

> You seem to know about their architecture. What do they do now?

Random 64-bit primary keys in mysql for newer videos. These may sometimes collide but then I suppose you could have the database reject insert and retry with a different id.

So a single cluster produces those keys? I thought it’s more decentralized.

With random database keys I would think they can just be generated at random by any frontend server running anywhere. Ultimately, a request to insert that key would come to the database - which is the centralized gatekeeper in this design and can accept or reject it. But with replication, sharding, caching even SQL databases scale extremely well. Just avoid expensive operations like joins.

Is the extra 64 bits simply used to lower the risk of collision?

> all of their video IDs are 11 characters, alphanumeric with cases

It's an int64, encoded as URL-friendly base64 (i.e. alphanumeric with _ and -).

Thanks for doing the maths - it does seem the sampling method would not be feasible. Taking the statistic of "500 hours uploaded per minute" and assuming the average video length is 10 minutes, we can say about 1.5bn videos are uploaded to YouTube every year or 15bn every 10 years. So it seems likely that YouTube has less than 1tn videos in total.

If there are N IDs to draw from and M videos on YouTube, then P(ID used) = M/N if the ID is drawn from a uniform distribution, and P(At least one of K IDs used) = 1 - (1 - M/N)^K (not accounting for replacement).

If M ≈ 1e9 and N ≈ 1e18, and you sample K = 1000 URLs, then it's about one in 1e-09 that you hit a used ID.

Reply on news.ycombinator.com