Hacker News Reader

Comment by skrebbel | Hacker News Reader

I really like the ideas behind ksuid (near the end of the article). However, two quotes:

> Those concerned with UUID collision in a properly-configured system would find their time better spent pondering far more probable events like solar flares, thermonuclear war, and alien invasion on their systems.

And then further down:

> A “custom” epoch is used that ensures >100 years of useful life.

Wait, so the last 128 bits of a KSUID won't get me in trouble before the sun explodes, but the first 32 bits (the timestamp) will cause trouble well before my grandkids die?

I really wonder why they didn't reserve some more bits for the timestamp, if necessary at the cost of some less randomness. Could've made this stuff last for millenia at no extra collision risk, in practice.

View on HN

Yeah, which is why I'm toying with using ulid identifiers, which won't face timestamp issues until around 10900 AD. Source here: https://github.com/alizain/ulid/

Related: https://github.com/oklog/ulid

The repo in the comment parent to yours has a list that includes the one you mentioned and several others in various languages :)

https://github.com/alizain/ulid#implementations-in-other-lan...

Yeah. Not sure why they didn't just go straight for 64bit timestamps (maybe 10^-8 second granularity ~= 5.8k years) and 64bits of random data.

I also wonder if base58 would have been a bit nicer. base62 is of course slightly more compact, but base58 is nice that it reduces visual character ambiguity.

The original UUID's (from Apollo, as used in OSF DCE, e2fsprogs, Microsoft, etc.) used a 64-bit timestamp, with 100ns granularity. It uses as the start of its Epoch the beginning of the Gregorian Calendar (00:00:00.00, 15 October 1582), so it's good up to 3400 A.D. before it hits the 2038 problem. :-)

Note that UUID's, like IPv6 addresses, are sufficiently long that if users are needing to interact with them directly, You're Doing Something Wrong. So the whole base-62 versus using hyphens versus base58 discussion misses the point, in my view. Computers will generally be exchanging them in 128-bit binary format, and they should only really be dumped out for debugging reasons.

  > Note that UUID's, like IPv6 addresses, are sufficiently 
  > long that if users are needing to interact with the 
  > directly, You're Doing Something Wrong

One would think so, but you would be surprised how many times someone has typed out to me (or copy/pasted) a UUID for debugging purposes, or how often I have had to eyeball UUIDs in production logs.

We eyeball, compare and copy/paste git commit IDs all the time, and they're longer than UUIDs. I don't see any problem with it as long as there's good tooling to help us interact with these IDs (e.g. git allows us to use the first few characters of a commit ID if there's no collision).

[Author here: Hey there Theodore -- first of all thank you so much for that other comment with first-hand knowledge about Apollo => UUID. This is the kind of priceless nugget that makes HN such an interesting place.]

From a systems perspective I can definitely see why one could reach the conclusion that if users are interacting with them directly, they're doing something wrong.

What we've seen is that IDs are routinely copied between different systems for debugging purposes. An example at least for Segment it is common to have IDs pasted into support tickets, and the search engine that indexes these will tokenize on the "-" in a UUID. The difference between finding the solution in minutes vs hours could be that two systems can be correlated by searching using the UUID. Leaving these out is a minor tweak that pays dividends over time.

For anything human-exposed (and in JSON or URLs) we use the 27-byte base62 encoding. In the database we store KSUIDs in their 20-byte binary (base256?) encoding.

The universe is a few billion years old, and in all likelihood will last at least a few billion more.

I wonder how people manage to keep a straight face while calling something "universally unique" that can't even cover a time period corresponding to the recorded history of our own species. Perhaps it never occurred to them that archeologists and paleontologists might also find time-based UUIDs useful?

Space is cheap now. We don't have to fit everything in 128 bits. We could just use good ol' 64 bit time_t, add another 32 bits for nanosecond granularity, throw in a decent amount of random data, and still get identifiers somewhere between the size of a git commit ID (160 bits) and a bitcoin transaction ID (256 bits). Nobody writes these things by hand anyway.

Archeologists actually have their own versions of UUIDs and massive books/databases cataloguing them. Since dates are constantly subject to revision though, they don't generally use timestamps.

The very fact that scientists have had to come up with their own identifiers is testimony to the woeful inadequacy of what we're calling "universal" identifiers.

Archeologists of the year 8017 might also thank us if we were kind enough to label each piece of our data in a way that makes sense to them, including precise timestamps. Heck, they might still be using the same format if we did come up with something truly universal!

We've been using roughly the same formats for a couple hundred years already, well before computers were a thing. Problem is, that also means every government has its own slightly different databases that that no one wants to integrate.

I've rarely seen dates encoded because they don't provide much useful information. Sites themselves rarely have firm dates and individual excavations get their own (infinite) ID blocks. Each feature and artifact within that excavation is tagged with the block, alongside a bunch of metadata and paperwork. Every artifact carries with it a stack of paperwork documenting where it came from and how it was buried, making shipping the things expensive.

These collections eventually get bundled into larger site or regional packages alongside computational models and research papers, adding yet another hierarchial later.

The whole thing is very bureaucratic and a bit crusty at times, but still very scalable for a system designed before computers were even imagined. As with most things, the friction comes from trying to move things between the physical and digital worlds, not the IDs themselves (which are just guaranteed unique tags).

That's because now matter how many times we're told to simply trust uuids we will always want to add a timestamp just in case :)

Maybe we should just admit that no amount of poetic hyperbole like "until an alien invasion" will quell our innate fear of having uuid that will collide.

I have heard people laugh at the risk of UUID collisions and thought we were safe but heard two years ago that a consulting company I worked with had seen real issues with UUID collisions on a production system.

I doubt many people would argue with the math behind modern uuid generation functions. Those of us that have been around for a while know the problem can come from the implementation and the storage for the UUID.

For example: Bugs in code that may generate a unique ID once and then use it multiple places.

Imaged OS installations that may cause a UUID to be duplicated

Spoofing another ID for malicious reasons

Accidental database update statement

People tend to give UUIDs magical properties but it needs to be treated similar to an unique integer.

If you are using uuid generation that uses MAC address and time stamp, then you can have problems with multiple processes generating uuid for the same system. There are other uuid generation algorithms though.

The 100 years thing is a peeve of mine too. Especially if they want to see this widely adopted.

That is how things like IPv4 address exhaustion and Y2K problem happen. It's 2017, add another byte to your timestamps.

Are UUIDs secure for creating a unique user session identifier?

No, UUIDs aren't meant to be cryptographically unique. Session identifiers should be generated by your friendly neighbourhood crypto rng man

If you're not specifically requesting a time based ID, any proper uuid lib should just return 16 bytes of randomness from the system rng.

Not quite, most generate v4 uuids which have a few fixed bits.

Getting pedantic, but a proper uuid lib should be returning 122 bits of randomness, and six bits of fixed information that specify the uuid version and generation method ("random").

For what purpose? Where's the real world reason and interop? Just because an RFC says to set some bits? Is there any code you're likely to hit in your app that actually cares?

I guess the purpose is to be able to put different UUID generation schemes into separate kind-of "namespaces".

That way, a UUIDv4 (random) will never collide with a UUIDv1 (timestamp+mac+...), which again will never collide with UUIDv5 (SHA-1).

So if one of the UUID generation schemes is flawed, these are easily filtered out, and moreover, will never interfere with any other UUID version/scheme.

You need to double check the library you're using. For example, older versions of the Python standard UUID library would fall back to an insecure RNG silently.

It's easier to just read 16 bytes from /dev/urandom and encode it in hex/base64 yourself for a random token.

The ones most modern UUID libraries generate (v4 UUIDs, variants one or two) have 122 random bits from the OS's csprng and 6 fixed bits.

It depends. I wrote a blogpost about the security implications of UUIDs back in February (https://blog.silentsignal.eu/2017/02/17/not-so-unique-snowfl...) and it shows that not all standard libraries make secure-by-default easy for the developer. I developed a MIT licensed plugin for Burp Suite (https://github.com/silentsignal/burp-uuid) that can help pentesters and security minded developers to detect insecure UUID versions, and even in that plugin I described version 4 UUIDs as "randomly generated, although [their] entropy should be checked".

I'm banking on ULID and surprised that they didn't adopt it, it's a much more approachable solution and battle tested.

Reply on news.ycombinator.com