Hacker News Reader

Comment by xenadu02 | Hacker News Reader

This seems like useless optimization that has a high probability of biting you in the ass later on.

Using the standard UUID generation facilities in your OS of choice there's zero chance you get something wrong and screw yourself.

UUIDs are great because we can pretty much guarantee global uniqueness. Acquire a company, decide to integrate with someone, need to merge a database, etc? No problem, zero chance of record collisions no matter what happens in the future. (It also means zero chance of accidentally interpreting record #58274 as type A when you meant type C).

Furthermore, a 1 in 1 million chance of collision is far too frequent for my liking, but even if it were acceptable what happens when your service/product becomes far more popular than you imagined and you blow through your initial estimates?

View on HN

This is absolutely right. I suspect some people do this because Twitter went out of their way to get 64-bit ids[0] -- but Twitter did this in large part because they made the mistake of baking that bit-length into the protocol back when they were tiny, and since they don't control all clients, it's very difficult to change.

For comparison, here's an eager.io URL:

https://eager.io/app/ZYBle8qUhKFJ

And here's an equivalent with UUID in base-64:

https://eager.io/app/b8tRS7h4TJ2Vt43Dp85v2A

It's a rare application for which those differences matter. (NB: I don't know anything about eager.io... it might be important for them!)

[0] https://blog.twitter.com/2010/announcing-snowflake

It becomes a stylistic choice, but I can say we weren't happy with the experience the longer ids gave users. It made most of the url a random hash of characters, not anything significant. The URL is as much a part of the UX as anything else.

"The URL is as much a part of the UX as anything else."

Given what we've seen from the browser vendors lately (hiding bits of the address bar, etc), I'd say there's a lot of momentum to disagree with that statement.

I think it depends on who your customer is / what your product is. When I'm developing against an api, I end up writing API calls in my terminal to test things. I prefer cleaner URLs for APIs.

Is the difference between bkirwi's two examples really significant? The eager.io URL is still a mash of random characters.

(FWIW I like seeing UUIDs in my URLs, because it tells me that the company on the other end has its shit together. That might be a bit programmer-specific though)

There's also impact at the data layer. Larger IDs make indexes get a lot bigger. Also, any article on UUIDs that mentions locality without even mentioning using type-1 UUIDs should be taken with a grain of salt.

We looked at using UUIDs last year for a project in MySQL since we had good use cases for handling ID generation outside of the data tier. We originally prototyped using type-4 UUIDs, but found the locality problem made that a non-starter (updating indexes seemed to get exponentially slower when tables went over 10m rows). But switching to type-1 UUIDs made that problem go away. Still, we eventually chose not to use UUIDs since 128 bits made our indexes too large. At the DB level, we represented them as BINARY(16), but even with the pain that caused, our indexes were still too large.

Perhaps it's just a MySQL thing (I didn't make that choice) but, from our testing, I'd be very wary of using 128-bit IDs and would probably choose something like Snowflake or other centralized ID generation service.

Sometimes you don't need to index all 16 bytes, and can get by with a prefix index.

Note that the 1 in 1 million figure is for 100 million IDs generated, that is, after 100 million generated IDs there's 1 in a million chance that two of them are identical.

It's all fun and games until you discover that one in a million is next Tuesday:

http://blogs.msdn.com/b/larryosterman/archive/2004/03/30/104...

I've watched as the application we develop at work has switched from ancient single and dual core Linux boxes to (now) dual-dodeca-core boxes and all sort of "completely impossible" races have surfaced. All progressively more "impossible" as the number of cores and their speed has kept on increasing.

The problem of both locality and length also only exist in modern databases. Good old Ingres QUEL offered the choice of hash and btree indexes, and the the ability to compress indexes. Index compression is of course much easier with sequential indexes, but it would at least optimize the static host part of the uuid away.

Yea, if you remove the dashes from regular UUIDs and base64 encode them they're not much longer than his custom ones.

> No problem, zero chance of record collisions no matter what happens in the future.

Well until 5236-03-31 21:21:00 UTC (assuming my calculations are correct) using v1 UUIDs and assuming unique MAC addresses and no race conditions.

Only until 5236?? We definitely need a better solution.

Assuming unique MAC addresses is a very poor assumption.

At one company I worked at we once received a huge shipment of ethernet cards (several thousand) that all had the same MAC address. We didn't realize it until customers who had purchased multiple pieces of equipment began to call in.

I've heard several similar anecdotes from other engineers at other companies.

Yeah, very bad assumption.

A lot of platforms (iOS) also spoof MAC addresses too

The spoofing happens when talking to other devices (not themselves); i.e., there is some layer of the device that knows its true MAC address and the UUID generation code should use that layer.

Can confirm that this is a bad idea as manfuarcturing defects do happen. Took a shipment of brand new NetApp toasters this year and they all had the same WWN/WWPN's!

Needless to say NetApp where quite embarrassed and we are glad that we noticed it in pre-prod otherwise it could have been quite bad.

I appreciate where you're coming from, but as it says in the article, UUIDs are really unnecessarily long for no good reason. The traditional hex encoding is also much longer than it needs to be.

In any case, the article exists to help you decide how long you need to make your ids for you to be comfortable. If you decide you aren't comfortable with a 1 in a million chance, the math is there for you to figure out what does work for you.

> UUIDs are really unnecessarily long for no good reason.

Their purpose is to be universally unique, presumably indefinitely. The practical choice is to align on a power of 2.

When UUIDs were invented, they were based on a MAC address plus a timestamp. MAC addresses are 48 bits, leaving 16 bits. Of these, between 1 and 3 are used up giving the variant code and another four bits are used giving the version code.

If they'd used 64 bits, there'd be between 8 and 11 bits left for a unique timestamp. Which is OK, but not a very strong guarantee for a "unique" scheme.

As it happens the length has made it possible to create multiple versions of UUIDs that are generated in different ways. UUID has been successful at separating the concerns of the structure of the ID from how it is generated. Most alternative schemes are effectively implementation-defined.

This is all true, and I'm not here to say the original implementers did anything wrong. As you say though, UUIDs are a general solution, which means they're not going to be optimal for every problem.

UUIDs are optimal for every problem, in that they are unique and fairly fast to generate. There can be no better solution than that, except maybe to save a few bits. And we're only talking about 16 bytes here; not much to complain about. They're cheap at the price.

Yeah its hard to get folks to come into the modern age of computering. FOr some reason there's tremendous resistance to abandoning the old crusty buggy solutions (indexes and handles that collide and need remapping) and using stuff that just works (identifying everything immutable with a uuid). Folks say 'its too many bits' and 'it takes too long'. Then they proceed to write megabytes worth of patchy code to try and save 64 bits worth of identifier.

As for database efficiency, that's a problem. But its a problem with the database - certainly no comfort for the user. But databases could do better, and I think they will one day come to handle uuids with less grief to the developer.

They already do. Postgres and SQL Server have UUID types.

Yeah, exactly. Building weakness into one's protocol because of an unhealthy attachment to saving a few bits and an even more unhealthy attachment to MySQL just seems foolish.

If you work with large amounts of data it hugely matters. 16 bytes times 1 billion records ends up being way more than 16GB of data, indexes, foreign keys, hot backup, etc. to manage and query.

Even that is a small amount of memory. I have that much RAM on my computer. See, its an obsolete conversation, yet folks raised in an ecology of scarcity can't get past it.

It depends on what you are optimizing for. If you need to store your entire database in physical memory, then the difference between 16GB and 128MB is significant.

No, but I think most people overoptimise on the shiny stuff ("oooooh, I saved some bits!") over the important stuff (nasty surprises, several years from now).

Yeah, but sometimes "saving some bits" is really important. It's also really inconvenient to work with UUIDS, like in writing SQL queries etc. The solution these guys came up with obviously works for them.

Precisely the argument from Dilbert's Y2K episode: http://www.tv.com/shows/dilbert/y2k-12015/

How are these eager.io IDs any easier to work with in SQL queries etc.?

> It's also really inconvenient to work with UUIDS, like in writing SQL queries etc.

http://www.postgresql.org/docs/8.3/static/datatype-uuid.html makes it look pretty easy…

UUIDs are 128-bits...

> If they'd used 64 bits

Actually I think that is the biggest problem with the article. It gives you no way to asses which probability is good enough.

I think thats a business decision, not a technical one. You have to decide what odds you are willing to accept, based on how many records you anticipate, how hard it will be to increase the length if that estimation is wrong, and how bad it would be if there was a collision.

In any case though, there are certainly many bad things which could happen to your company which are more likely than 1 in a million.

Reply on news.ycombinator.com