You Don't Need UUID
henvic.dev
henvic.dev
I would challenge the premise we appear to be starting from, that the average end user cares to be dealing with any random string of numbers and digits. GUIDs work well, they’re implemented everywhere, and you won’t find out long after you go into production you made some mistake that is going to make it so you have to migrate away from them.
While somewhat less dense, base58 or rfc 4648 base32 mitigate these issues.
My own preferred format for UUID7 is one I call "id25". It's really Base35 because alphabets of 35 and 36 characters both need 25 characters to represent the 128 bits in a UUID. So I can start with Base36 and take out one of the next most ambiguous character pairs. The result looks something like '0pydgw5pifvapk5zyhmpso5tx'.
The two other advantages of using this id25 format rather than UUID7 are:
- The id25 format is quite distinct from UUID4. So if you have UUID7s being generated distributedly (rather than centrally) and the UUID7 time-ordering feature is important, it's nice to have a format that a trivial "if '-' in uuid_string..." check will spot.
- Because there are no hyphens or symbols, the whole id25 uuid can be selected in web page with a double click. Whereas a uuid will need a mouse movement to get all 5 parts.
Most of the use cases I can think of where I'd want an ordered UUID, I probably wouldn't need to lexically compare the encoding of multiple such IDs (where I can see plenty of utility in the ability to compare the underlying IDs).
-;cAU'UX5;VnaA:`a2[1Sometimes when I order parts for my car the site wants the VIN to verify that the parts are correct for that vehicle. I often have to make two or three attempts at entering it to get it right.
[0] https://bohwaz.net/archives/web/Bubble_Babble.html
[1] https://iancoleman.io/bip39/
[2] https://web.archive.org/web/20090918202746/http://tothink.co...
Although after reading that Babble link it sounds like a great idea for product id's. Scenarios where it is unlikely for people to have to read it out but when they do it's easy enough to convey.
I regret that there are scenarios that we should have converged on it but didn't.
A developer, which I took to be "the user" for the purposes of this writeup, cares. My small concern is that the rand is insufficient to generate a unique enough string (just use a lib like snowflake to get overkill entropy), but I'm sick of having the format at all for inconsequential ids (eg https://www.uuidgenerator.net/).
It’s a small gripe, but I need to copy IDs multiple times per day and it adds up. Other ID formats like cuid or KSUID don’t have hyphens in their canonical representation, and it makes them far more pleasant to work with
if not it might just become a situation of optimizing for your own workflow. and considering how ubiquitous uuids are and how common dash delimiters are outside of that(jira ticketscome to mind) maybe it would be better to fix places where double click selections do not include dashes instead of trying to avoid dashes in identifiers?
ULIDs don't have this problem, for the record.
Does it make the URL in the URL bar longer? Yeah, but does that matter?
UUIDs are at least visually kind of white noise, companies have no problem+with+links?that=look&like=this, so why do they care about UUIDs?
You can't beat that. Or would you rather receive texts that include the following?
https://www.amazon.com/dp/B09V3HZ8B5/?pd_rd_w=DAycy&content-id=amzn1.sym.04e31e32-5e01-4048-8b59-a557891d595f:amzn1.sym.04e31e32-5e01-4048-8b59-a557891d595f&pf_rd_p=04e31e32-5e01-4048-8b59-a557891d595f&pf_rd_r=M1KXM468KNZJV189JBE7&pd_rd_wg=3O1uC&pd_rd_r=e51c1edc-213f-4c6d-b081-e23afa6bf4ab&ref_=vn_s_iwp&qid=1694453553
Let's not match the lowest common denominator, shall we?
As you have been told apple.com/iphone = www.amazon.com/dp/B09V3HZ8B5
The rest of what you posted on the Amazon link is tracking bullshit. Now, THAT is irrelevant !
Amazon product URLs are looong, but maybe people don't share them enough for Amazon to care. YouTube and Twitter put more effort into making them short.
So you could have urls like:
example.com?<a single icon>&productname=somethinghumanreadable
A more "apples to apples" comparison (pun intended) would be the MagSafe charger:
https://www.apple.com/shop/product/MHXH3AM/A/magsafe-charger...
I'm sure they put some thought into not making it ?v=f6dde15a-3485-4f36-b0b9-a3b56fbc12ae.
Yes. That's one of the author's arguments.
) A simple ID like 3c6n63N is more than enough to represent any product while keeping it readable and making communication easier. A UUID alternative like a73ba12d-1d8b-2516-3aee-4b15e563a835 is just wasteful from an user’s perspective.
"Does it matter" is a very reasonable response to someone expressing their opinion.
If I say "x matters and here's why" and you respond "yeah but does x matter?", you're either not paying attention or being rude and dismissive.
And what do they mean with communication? Reading it aloud? If the URL is too long to stick in a tweet or message, there are plenty of URL shorteners out there.
On link-sharing sites like this one, we care about the URL’s we share and will often strip off the tracking stuff.
And I bet that for most users 3c6n63N is as confusing as a73ba12d-1d8b-2516-3aee-4b15e563a835
You don't need to make it a user exposed identifier, and you can have a centralized (but out-of-the-critical-path) process assigning friendly IDs if you can afford visibility delays (for either the whole item or at least the friendly ID, which if it is the only user-facing locator may mean that the individual item can’t be directly accessed even if it is visible in aggregates, until that is assigned; if you allow access by both the UUID and the friendly ID, with the latter presented as preferred once available, this is resolved but this may appear messier.)
Others have noted the usefulness of generating ids for user-created data without involving the server, while being able to post that data at a later time without worrying too much about collisions.
Suspenders-and-belt practice would of course dictate that the server do some type of collision detection and mitigation before storing the received data.
If UUID exists in database and user ID is different, tell client "Yo, we've got a collision here. Your new ID for this data is blah".
Generate new UUID. If I've ever used that UUID before, generate a new one.
If the server rejects my UUID, switch to the new one provided by the server
Personally, I'd look on home-brewed solutions for generating unique "friendly ids" with the same deep suspicion I look at home-brewed crypto, particularly if this were done client-side for multiple clients without involving a server round-trip. Getting it right is a Hard Problem. The perceived complexity of UUIDs is there precisely because it is a Hard Problem.
No, it doesn't.
If you're using a central system to generate id's (UUID's or other), then you are indeed missing a lot of the benefits of UUID's and might as well use some other ID scheme.
The benefits of UUID are many systems can generate them, and your database can eventually accept them, all with a very high guarantee of uniqueness/no-collisions. Not to mention the other nice bits like non-sequential and non-enumerability.
Most modern databases have UUID/Binary types/functions to allow for more efficient storage/handling anyway.
Nowadays many links are much longer because of referrer information and tracking parameters. So the UUID doesn't add much.
Not that people will care. Once a URL is longer than a few words plus a code most users won't even look, even if they bother that far. You'll be in danger of starting to hit some browser/server/proxy limits though.
Not that I mind that sort of information being lost, I'm happy for the incessant tracking of everyone's everything these days needs to go to hell.
Address bar: https://www.youtube.com/watch?v=SHWhKRKlG98
Share button: https://youtu.be/SHWhKRKlG98?si=gwQHHze3UnjTobJg
I'm really reaching here, but an ID that omits special characters is easier to extract from a url, since it can be double-clicked in a URL bar to select it, whereas a GUID with hyphens forces the user to select the beginning and the end of the string.
Realistically I think this matters to developers more than users (I hope your application doesn't force users to interact with GUIDs in the address bar) but that's the only "waste" I can identify.
EDIT: Someone elsewhere noted that they can base64 encode their GUIDs to achieve this and I'm seething that I'd never thought to do that
(Maybe that's what you meant, amd yes, it's still clunkier than simply double clicking, but not much)
Maybe the double-click logic needs to be less stupid then. Or have it progressive, e.g. double-click = alphanumeric, triple-click = stop at symbols, quadruple-click = stop at slashes only, quintiple-click = the whole URL.
Use the device microphone to listen for swear words and adjust algorithm with machine learning accordingly
I usually did that in .NET simply because I thought it was prettier without the hyphens.
The is-hyphen-part-of-a-word issue depends on the text control (browser, UI widget library, …) / terminal you use, which is not universal.
In any case, the hyphens/dashes are not required part of the representation, and one can use the “dashless” representation (exactly 32 hex digits) in URLs and in-code without any issues whatsoever, as far as I know; for example, Postgres accepts either “dashful” or “dashless” UUIDs the same.
It might, e.g. with embedded systems. I can neither confirm nor deny that we once BOFed our own devices because we assumed that 1200 bytes was a big enough buffer to hold a URL.
I appreciate shorter URLs any time I copy and paste them, which always involves looking at them and sometimes involves scrolling to the end to remove tracking- and search-related fluff.
128 bits is an absurd amount for a unique ID within a single system. Even 48 bits is very, very large -- enough to provide a unique ID (MAC address) to every Ethernet device made for something like 100 years.
The purpose of a UUID is to provide a negligible probability of collisions between randomly-generated IDs in a global space forever. Almost no applications require that, so most of the time using a UUID is just needlessly taking up space.
What you want is that the number of possible items to enumerate is significantly less than the square root of the cardinality of the ID range, means you can relatively safely randomly generate IDs with few collissions.
You've never needed to give an order number, member ID, account ID, invoice number, etc?
I agree. If there was a shorter standard, I would use that for most cases.
My title is too presumptive. Yet, the reality for almost everyone who doesn't interface with other systems that already require UUID is that.
The brief article gives some points – backed by trustworthy sources – and shows some alternatives without further ado. I didn't spend days writing it. I could cover many more scenarios but didn't invest my time in it. So what?
Not the person you're replying to, but the end of the first paragraph in your article is
> and I want you to understand why you *certainly* don’t need it
(emphasis mine)
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
When one wonders why that was included they might discover that the "short ID" example given near the top of the post is an Amazon link. One might then further wonder what the purpose of this article is.
[0] https://github.com/jetpack-io/typeid [1] https://github.com/ulid/spec [2] https://github.com/segmentio/ksuid
I really think when you're picking PKs, you should simply use whatever the DBMS recommends for performance. It's not the PK's job to be typed, sequenced, human-readable, or anything like that; that can be handled by other cols and logging rules. Its job is to be fast in those joins you'll constantly be making against it.
It also matters, albeit to a much lesser extent, to others. Postgres stores tuples in a heap, but the PK is still a B+tree (ish), so an INSERT or UPDATE heavy workload will suffer somewhat.
The good thing about UUID is that it's omnipresent. From what I've heard, it's this lengthy (2^32) because it was hard to guarantee uniqueness when it was conceived in the telecom industry. The length is overkill, and per se, that's fine, but the fact that it dampers communication is awful.
That all said, since posting this, I've come to terms with accepting that it's part of life ¯\_(ツ)_/¯
P.S. Using a second human-friendly ID to end-users is an alternative adopted by some projects. However, most projects don't bother, and also, most good IDs you might want to share with people would make UUID unnecessary anyways (in practice).
And if you want to be really human friendly, use one of those silly name generators.
UUIDs are 128 bits, so it's 2^128 rather than 2^32
Factually, X != hash(X). Sometimes you can make the simplifying assumption that X == hash(X), but only in well-defined contexts, subject to proper risk analysis; never in general, or as a presumption of a system that needs to be correct.
Yes, UUIDs are overkill for most applications. But CPUs and hard drives are, relatively speaking, cheap. Using an existing, battle-tested unique ID library implementation has advantages. The 8 bytes per record you're saving over bigserial is, for most use cases, negligible. 1,000,000 rows? You'll save 8 MB by switching away from UUIDs.
Most databases won't be that large. Use a UUID if you want; pretend you'll have Really Big Data some day if it makes you happy. Render it using a special function if the hyphens are too ugly.
The argumentation in this article is pretty poor from my experience. A UUID isn’t meant to be handled by the non-technical end user. The end user usually doesn’t and shouldn’t care about the URL. I can assure there are bigger architectural problems in your design if your user has to care about accessible internal ids.
https://github.com/a73ba12d-1d8b-2516-3aee-4b15e563a835/e16957bf-7d7f-41f1-97f4-98c93a6c3540/issues/4e4e9133-bcdb-45b9-8dfe-9e951846e3c6 https://github.com/4e4e9133-bcdb-45b9-8dfe-9e951846e3c6
urn:uuid:4e4e9133-bcdb-45b9-8dfe-9e951846e3c6
Both of those solutions typically make it hard for a DB if you write new entries, assuming you have an index on the ID.
In addition it might be more calming to actually be sure that a particular ID is not in use without doing a round-trip.
Is it practical to pre-allocate empty entries and reserve a set of them?
This is only a problem with extremely big indexes.
Now in many, typical applications you'd have to scale up quite significantly before this becomes a problem.
But if your application requires you to store small, new entries very quickly, you'll start to notice this even with moderate scale. Disk persistence is often the bottleneck already, and this might make it worse.
One fun and useful reference: https://devblogs.microsoft.com/oldnewthing/20190426-00/?p=10...
It's particularly interesting that Microsoft SQL Server was designed to optimize indexing for UUIDv1 machine IDs. That makes a certain amount of sense for a database cluster if IDs are sorted by machine. Of course, developers don't let developers use UUIDv1 in 2023 because those machine IDs are not secure in a general sense and can be a privacy/data leak in the worst cases.
Other databases sort/index UUIDs differently. There's no real "standard" and optimizing the storage of a UUID key is a game of playing to the strengths of your specific database.
(On one project I put some work into matching the much-better-defined ULID sort order to MS SQL Server uniqueidentifier columns for better database locality.)
Lots of languages are catching up to UUIDv7, which solves the indexing issue.
Nitpick: Google isn't concerned about the discoverability of private videos; those can only be viewed when granted access. You're thinking of unlisted videos.
Don’t forget: UUIDs are not dash-separated strings. They’re integers. You can render them differently if those dashes are sucky.
I like its simplicity. It is sequential. It has a very low probability of collision. And it is of a more reasonable length.
Because it encodes the time, theoretically you could use it to grab the CreatedDate of a record without a need for another field.
In particular databases (where UUIDs taking space was a big concern) they have largely switched to a packed binary format that makes the size of UUIDs over time a non issue for all practical purposes.
If you have to look up by UUID from outside and have no foreign keys, UUID PKs can be faster. But that's the classic use case for a non-relational DB.
https://en.wikipedia.org/wiki/Snowflake_ID
They fit within 64 bits, allow for more than enough processes to handle 10k+ transactions per second, give enough of a timestamp headroom for decades into the future, and where ID generation can be made isolated to each process.
They don't work well for anything related to archival work, but you might as well use a regular ID for that anyway, unless you're also actively scraping terabytes of data off of the Internet every second, in which case UUIDv5's good enough for your extreme edge case.
...But at that point you might as well just roll your own 128-bit version of a Snowflake ID.
Or UUIDv7, which is designed to solve the same problem
They will be in a deterministic order, but will appear semi-random to the end user.
For things like product IDs or user IDs, etc you don't actually need them to be random. But perhaps you don't want them to simply start counting sequentially.
1. Compact sequential is used, obviously, where order matters. It has the drawback of requiring a coordination with a singleton. (This can be sharded/vectorized, of course.) Aside from that, it also leaks the number of objects/transactions, just by looking for the highest number available. Can be varint-encoded very nicely.
2. Compact non-sequential is used where I need a small identifier, but not leak the number of objects. Since it's compact, I must still guarantee uniqueness as in (1). This is currently implemented using a block cipher on top of the compact sequential ID generator. The drawback of this is that the domain may still be in guessable territory, depending on how much is generated. A 64-bit integer filled with 4B only requires 4B guesses to hit a collision. I don't use this much. The key can never be rotated: a key number would eat up precious bits.
3. Sparse random, used where non-guessability is important, aside from not leaking rates/counts. Take a Google Docs sharable link as an example. I doubt YouTube cares about this. This is where something like a 128-bit number like UUID or ULID shines. The space is large enough that uniqueness is assumed, given a decent PRNG.
Sure, I try to use the nicest one at any given point (e.g. using a compact sequential instead of non-sequential during debugging.) But fact is that sparse random just tick more boxes.
4) Sparse, human readable. For "vouchers". They are bearer tokens that give requests more powers, e.g. to create an account or act as admin. These should be reasonably human readable, so they can be spoken. They obviously need to be sparse and hard to guess, which requires a trade-off in length.
I present them in three ways: base32, english words and QR-code. Pick one; they all do the same. For copy-pasting, base32 might be best (or base58, by all means.)
For shouting to a colleague, the sequence of english words might be better. I'd add other languages as needed: it's just a fixed list of words. The nice thing is it can encode the sequence in base-500 or base-1000 without being obnoxious. (The Matrix protocol and others use emoji lists, but it's the same idea. [1]) Finally, if you have a phone in your pocket or camera on your computer, perhaps the QR-code is the easiest way to use the voucher code.
[1] Actually, IIRC, Matrix only uses 64 emojis, which feels a bit wasteful.
not sure if this is duplicative but down the road i figure I can navigate whichever direction i need to go in
https://colab.research.google.com/drive/1ec4n7Ex9bnkl_c45EUl...
- Don't invent your own ID datatype (especially 11 byte one), this is almost guaranteed to cause dangerous bugs, because each integration will have to carefully implement/hack it.
- Use UUID (preferably the new v7). 128-bit UUID is implemented, for you, pretty much everywhere.
- Serial integers still work too, but you should choose them consciously to fit the data model.
- Implement "natural keys" if you want pretty/memorable/Cool URLs. Never use non-standard PKs to store custom semantic data, because inevitably you will get garbage PKs that need to be fixed, and migrating a PK value is extremely risky.
We could encode the 128 bit UUID integer in base58 as well if needed. Its textual representation won't conform to the standard but we'd get the same number of bits. Which would be 2^122 or 2^121 bits of uniqueness not 2^128 if it's a proper UUID.
11base58 character certainly doesn't have the 2^122 bits. So we could decided separately if we could either reduce the number of bits needed and/or use a different encoding.
> If you click and buy any of these from Amazon after visiting the links above, I might get a commission from their Affiliate program.
:-)
A UUID can also be encoded in any form, it doesn't need to be represented as the dashed string notation which is common, you can just as easily use the base58 alphabet suggested in the post.
But the code in the post doesn't encode to base58 correctly. You need to map 58 bits of the input, sequentially, to one character in the alphabet as output. You can't just mod each byte of the input by the alphabet length and use the corresponding alphabet element.
Since the alphabet is 58 characters, each character contains log2(58) bits. Since that's not an integer, the encoding process is a bit more complex than just mapping bits to characters in a table.
I agree lines 12-14 doesn't do a proper encoding, but that's not what I was after. Would you still say it's still wrong if you consider that my function is not really encoding the string per se, but using rand.Read to generate entropy for what I want to be the final output (random string with the base58 alphabet)?
Looks like the most used base58 package is https://pkg.go.dev/github.com/btcsuite/btcutil/base58, but looking at the implementation [0] I'm not impressed, there's definitely a much better approach.
[0] https://github.com/btcsuite/btcutil/blob/v1.0.2/base58/base5...
But how you encode 11 bytes of data is kind of orthogonal to the important thing, which is that you have 11 bytes of data. Those bytes should be always be stored in memory (in your application, or in your DB, or wherever) as the actual 11 bytes of the ID, and not as a base58 or base64 or JSON or whatever other kind of string that can be decoded to the actual 11 bytes they represent.
Likewise, a UUID shouldn't be stored as a string like "64d3f2e0-a4dc-48d3-98ad-7f09eb3b082f", that's a specific encoding of the actual 16 UUID bytes, you should store, process, etc. those bytes directly.
> 6 predetermined variant and version bits, leaving 122 bits for the randomly generated part
> --wikipedia[0] https://trustoverip.github.io/tswg-cesr-specification/draft-... [1] https://keri.one/keri-resources/ [2] https://trustoverip.github.io/tswg-keri-specification/draft-... [3] https://github.com/WebOfTrust/keripy/ [3] https://trustoverip.github.io/tswg-acdc-specification/draft-...
1. When dealing with user friendly IDs, it's often important to make sure there are no ambiguous characters. This is a UX requirement for anyone that may need to read an ID and type it in at any point. For example, this means removing certain confusing characters like "0oIi1l5sS", etc. You end up with a much smaller set of characters. In my case, young children were required to type in teacher codes. Needless to say, I had to limit a lot of possible confusing characters.
2. You will have collisions. How do you handle them (ie, retry N times until you get a valid one)? What happens if you can't get a valid one? This happened in a product of mine. We had ~8 character codes that were human readable and type-able and we... ran out of codes.
3. How can you embed more information into the code such as versioning? It's often useful to prefix a code with some info in the event you need to modify it later (such as expand the character set or length)?
You can shrink the range by using a smaller prime number and throwing on out-of-range input. And then you can offset the output to make sure the base-N encoded value is always D digits long.
But besides human friendly slugs, which usually have an ID mapping behind the scenes in my experience, it seems like there might be more work than value for startups in many cases.
Databases and lots of other things also have native support for them which makes them even more appealing to use.
Of course, there are plenty of reasons to avoid them like the article discusses. I'd almost certainly not criticize someone for choosing smaller IDs as long as security isn't an issue.
I like using GUID specifically because there's usually a built-in implementation everywhere, and because it's so stupidly huge, the likelihood of a conflict is statistically zero, meaning I don't need to bother synchronizing against a server.
Yes, it's technically possible I could save a few bytes of bandwidth or something by using some kind of base encoding, and maybe in some kind of embedded case that might matter, but for a vast majority of cases, GUID is absolutely fine, and since it's used everywhere, it's also very thoroughly tested by multi-billion-dollar corporations meaning that I don't have to worry much about any issues.
So no, you don't need GUID, you don't need a lot of stuff in the software world. I could poll bytes from /dev/urandom and probably get something that works well enough, I could turn off garbage collection and just pre-allocate all my memory before hand, I could avoid an OS entirely and write straight to the metal, I could do a lot of things, but I don't because I value my time . GUID solves a specific problem pretty well.
EDIT:
The reason I'm specifying this is because I think the title would be better if it was something like "You (probably) don't need UUID".
That's a terrible way to use it is all.
We have different ideas what 'microservice architecture' means, I guess.
One of the key points of UUIDs is to be able to generate (probably) non-conflicting values without coordination
For example, you can update a postgres integer 1000s of times per second, sequentially, and 100s of 1000s of times per second if you allow for gaps/ unordered sequencing. The reason to choose a UUID is because it doesn't require a db. Totally fair. But you probably already have a db lying around if you're generating UUIDs to begin with?
Or because you need an unguessable token, which a UUID is great for.
But if you have a postgres db around somewhere consider just having a counters table and using that. It'll be fast enough for almost anyone, it'll return smaller tokens, and it'll return tokens that are sequentially allocated (in range. These are nice properties to have!
Unguessability is a nice layer to have in your security strategy, especially if you're passing IDs to and from a web page, but if that's all you rely on you're not doing enough.
Yep, the docs from Postgres are very clear that `serial` has gaps. But you can also just create a counter table and manually manage the sequence if you have strict serial requirements.
Of course, if you have serial requirements a UUID would already not be viable.
UUIDs represent a trivially easy way of implementing a non-incrementing ID with a ridiculously unwieldy address space that makes it supremely unrealistic for the vast majority of users to mess with. They’re just going to give up long before the first successful hit.
https://devblogs.microsoft.com/oldnewthing/20040211-00/?p=40...
id UUID DEFAULT gen_random_uuid()If you need something user friendly, use the first 6 or 8 characters.
If you need to worry about index performance due to enormous index sizes, you probably know it.
- the system time
- a process-global singleton counter (incrementing with each ID)
- a "salt" derived from some number of bytes taken from an entropy source
- a "fingerprint" of some number of bytes taken from the same entropy source
this is an awful lot of work to be doing for each ID, to what end? well issue 7 [1] says that
> Cuid2 is good when you need something secure, extremely collision resistant, and your system is distributed or decentralized (e.g. you want to be able to create records with ids on the client side), or you are building software that may need to scale horizontally).
but I'm not sure how cuid as implemented provides a better answer in any of these dimensions to a plain old UUID
maybe I'm just being cynical, I dunno
[0] https://github.com/paralleldrive/cuid2/blob/fb07094487ba5ad0...
Small correction here but it is not 128 random bits entirely: 6 bits are reserved for the version marker.
How readable is base58 in Arabic or Chinese? I've concluded that only the standard numbers 0123456789, algebra symbols +-/*= and phone symbols #* are universal enough for a global encoding. Any other insights are welcomed!
For instance, UUIDs allow mobile apps to create entities while offline.
0-9 and a-f are unambiguous and widely understood. They’re more “human readable”.
Until another standard similar to what the article is suggesting becomes widely implemented in standard libraries then uuid isn't going anywhere, although in principle I agree with many of the arguments presented.
AKA I will make broad sweeping stupid headlines
...because I am going to keep using uuid