Hacker News Reader

Comment by pgaddict | Hacker News Reader

It is not a matter of a couple milliseconds.

The loss of locality for reads is bad, especially for data sets that don't fit into cache / RAM (while the active set would).

Where it really bites you is writes, because it can trigger pretty massive write amplification.

Imagine you have 128 GB index on UUID column, that's ~16M pages (8kB) and insert 1M random values. Congrats! You've probably just wrote 8GB to the WAL, because of FPW and stuff like that. With serial IDs we'd write a fraction of that. It doesn't take much to hit max_wal_size and trigger a checkpoint, starting a new cycle with FPWs. Got a replica? Well, now you need to send the WAL over network. Is the bandwidth limited (another DC?), sorry to hear that. Is the replica sync and you have to wait. Well, that's unfortunate.

In other words, the lack of locality seems like a detail but at scale it's actually a damn huge deal.

View on HN

If you want locality, speed, simplicity, etc above all, use an incremented integer and be done with it.

UUIDs belong where you can't afford that simplicity. Where you e.g. cannot coordinate the creation of your primary keys. Or where you cannot allow them to be predictable. There you pay the price.

In practice I noticed that the size of PKs and their poor locality start to play a role only after a huge basket of lower-hanging fruit has been collected. There are relatively few places where their role is dramatic.

> UUIDs belong where you can't afford that simplicity. Where you e.g. cannot coordinate the creation of your primary keys. Or where you cannot allow them to be predictable. There you pay the price.

You often don't realize you you have non-simple needs until your application is reasonably mature, and in production. If you already picked integer keys, you now either forever deal with the issues caused by not using UUIDs, or you deal with the unknown-but-non-zero pain of converting to UUIDs.

The age old startup bargain: do you want to pay a small cost up front, or a large one later? If only you'd seen it coming!

> Where you e.g. cannot coordinate the creation of your primary keys.

You are writing your data into a DBMS. Coordinating the creation of primary keys is one of the cheapest tasks around, if you can't do that, how is your database still online?

> Or where you cannot allow them to be predictable.

You don't need to export your PKs for the rest of the world. You can have non-predictable data outside of your PK. Yes, a different column will still have some of the problems with index maintenance, but it becomes a much smaller problem if only one table cares about the value.

You can almost always opt for artificial PKs that are more performant.

But sometimes you have to make these keys public, e.g. as user or other resource IDs. You want to make them UUIDs so they won't be predictable. Not having to join everywhere with the UUID-to-artificial-PK table may be a bigger performance win than the losses from larger size of UUIDs.

Sometimes you have a distributed / sharded system, and don't want the keys to clash, and also avoid assigning ranges. Sometimes you have to accept someone else's ID, not originating in your system. In cases like that, large random numbers, e.g. UUID v4, work reasonably well.

Of course when you just have one DB, and a relative slow stream of new rows, it's easy to fully control PK creation. And this covers the majority of practical cases.

We use a macaddr8 that embeds a wall-clock timestamp (so they're ascending order, achieving data locality) with some additional shard and sequence-number bits. It's worked really well for us:

https://github.com/estuary/flow/blob/master/supabase/migrati...

we use macaddr8 instead of bigint, because it has a postgres serialization / JSON encoding which lossless-ly round-trips with browsers and it works well with PostgREST. The same CANNOT be said for bigint, which is a huge footgun.

Out of my scope, but why are UUIDs even discussed? ULIDs ~~(and i think Nanoids?)~~ don't suffer these same problems. Locality and ordering alone make me[1] think ordered ULIDs (and friends) are the only thing worth discussing.

Is there some value to UUIDs over ULIDs that make these discussions largely revolve around Autoincrement vs UUIDs rather than Autoincrement vs ULIDs(and friends)?

[1]: Again, totally out of my wheel house.

edit: Apparently UUIDv7 exists, which is similar to ULID, so my question pertains to UUIDv5 and below, i think.

edit2: I think Nanoids do suffer the same problem. They're just small... i think.

ULIDs have made our MySql caches breathe a sigh of relief.

One place we're avoiding ULIDs (and other counters) is in publicly-facing IDs. Preferring random to help keep them unguessable. (say what you will about security-through-obscurity).

So we do ULIDs for private IDs. Random UUIDs for public IDs. Seems to work well.

KSUID's are have temporal-lexicographical order plus 128 bits of entropy, which is more than UUIDv4.

https://github.com/segmentio/ksuid

I wonder if just encrypting them for public usage would be enough here. Then application can convert between public and private representation at will.

Encryption is good until someone leaks the private key. Depending on the application, it may be very difficult to reassign all primary keys and foreign keys when that happens, especially if they have already been used as canonical identifiers.

You don't have such issues with randomly generated ids, unless someone obtains a full dump of your database (in which case you will have to think about bigger problems anyway).

I’ve worked with ULIDs a bit but honestly haven’t operated at a scale where you might run into issues. And the main reason for choosing ULID was because I could only find experimental support for UUIDv6 or UUIDv7, and KSUID (another alternative) only had time precision down to a second.

And the reason for that was to have lexicographically sortable IDs (even if not monotonic, which would require an extra server) so we didn’t have to index a timestamp column for chronological ordering. The alternative was to have a traditional integer sequence but it’s not always ideal to expose those on your API.

Out of curiosity, why do your keys need time precision less than a second? Are they carrying some secondary expectations beyond uniqueness and k-sortable?

Uniqueness and the context is that we were modelling a chat conversation, so the limited precision presented issues with messages occasionally appearing out of order.

That also applied to our test suite so the choice was to use KSUID but still index a timestamp for accuracy, or just use an alternative with more precision.

What would you use for that application if you had to choose today?

Postgres supports UUIDs natively

Is that worth the pain of dealing with randomized inserts? I guess i just don't mind creating a ULID (or i guess UUIDv7 is newly proposed and sortable) and inserting that.

Native DB support is irrelevant to me for randomized bits unless it affects storage, sorting, paging, etc. Does it?

It does affect storage and sorting. A native UUID type uses 16 bytes. The alternative is text encoding (32 bytes for hex), or maybe a raw BYTEA. Postgres also has SortSupport for the UUID type, which basically means if the first 8 bytes only has one matching row, then the remaining 8 bytes can be skipped. Combine that with a ULID where the most first half is basically a timestamp, you'll get performance close to using a single 8byte BIGSERIAL.

You can also write a plpgsql function to generate these ULIDs in the database.

ULIDs are byte-compatible with UUIDs, so the only thing Postgres's native support gives you over ULIDs is that Postgres can generate new UUIDs for you instead of having to do it in the application before insertion.

It also buys you that support in the driver libraries.

I ran into this last year - storing them as UUIDs works great, unless all of the drivers for the language you're using (Go, in my case) try to cast/validate those bytes as a UUID before you can access them.

At scale you’re probably sharded across multiple DBs and you’re already operating through replicas. Point being you’re less likely to hit a warm cache as you scale up anyway as your application layer gets load balanced to different DB endpoints.

Reminds me of a talk about ZFS performance, where they presented some benchmark results showing that as the number of concurrent clints doing pure sequential IO increases, the more the load appears as random IO to the filesystem.

So with high enough concurrent load it's effectively all random IO and that's the primary thing worth optimizing for.

Yes exactly, with proper sharding, raw performance is not as important; to some extent, you trade it away for improved concurrency. In fact, I struggle to see how one would implement sharding with auto-incrementing integers (you would get ID collisions for different resources across different shards/database instances); there needs to be a way to uniquely refer to resources across potentially multiple databases and UUIDs are one of the best ways to achieve that.

Auto-incrementing IDs simply don't scale beyond a single host so I don't see how they can be good for scalability. Too many devs conflate raw performance with scalability. They are not the same at all - In fact, high scalability often incurs a performance overhead.

We did auto-increment integers with multiple servers like 20 years ago: The caveat is that you have to know how many servers are in the set in advance. Each server increments by the population size, and their starting number is their position within the pool.

Not hyper scale, but good enough for failover or a 3-5 node setup.

Oh this is clever! I guess it’s actually very similar conceptually to a vector clock. (In that you have partitioned the space of natural numbers into K countably infinite sequences).

If you wanted to (practically speaking anyway) overcome the requirement that K is known, you could borrow prefix based counting from the p-adics.

IIRC that's what MySQL Galera is doing by default

    mysql> CREATE TABLE animals (
        ->      id MEDIUMINT NOT NULL AUTO_INCREMENT,
        ->      name CHAR(30) NOT NULL,
        ->      PRIMARY KEY (id)
        -> );
    Query OK, 0 rows affected (0.34 sec)
    
    mysql> INSERT INTO animals (name) VALUES
        ->     ('dog'),('cat'),('penguin'),
        ->     ('lax'),('whale'),('ostrich');
    Query OK, 6 rows affected (0.01 sec)
    Records: 6  Duplicates: 0  Warnings: 0
    
    mysql> SELECT * FROM animals;
    +----+---------+
    | id | name    |
    +----+---------+
    |  3 | dog     |
    |  6 | cat     |
    |  9 | penguin |
    | 12 | lax     |
    | 15 | whale   |
    | 18 | ostrich |
    +----+---------+
    6 rows in set (0.00 sec)

Or increment by more than you need. If you think you'll only ever have 5 nodes, increment by 20. Lots of space for expansion and in practice you'll hit the datatype limits just a bit earlier - you'd need to work around them anyways.

Hi/Lo algorithm also works fine for most cases

I respectfully disagree with the notion that sharding makes resource usage somehow less important. Sure, it allows you to overcome the limits that would apply to a single node, but if you stop caring about using resources efficiently (e.g. memory / disk / network bandwidth), it's not very different from building huge and expensive single boxes.

Also, I don't think the blog (or me) suggested going back to using autoincrement IDs. There are other (better) options.

this sounds horrible until you realise that a modern SSD will write that in about 1.5 sec.

That was just an example calculation, to illustrate the write amplification factor, of course. You can scale it up pretty arbitrarily.

I mentioned only WAL for simplicity, but it also has to modify and write out the index pages themselves, and write them out eventually. And that's going to be mostly random I/O. Flash storage is good at handling that, ofc, but if things are adding up like this ...

Not to mention you still have to copy the WAL over network to replica, or perhaps to multiple replicas. And if you have physical backups with PITR, you gotta keep all the WAL somewhere too.

sure but not many workloads are writing out million inserts per second.

Mmmhhh should that not be highly efficient for flash based storage?

Then just make a generated column that's a 32bit integer hash of the uuid for this particular case and create an index on that? Use that during expensive queries that blow up your cache locality if it matters.

A 32bit integer hash won't have locality either.

Ok. Pick a resolution where you won't get collisions and that is a native datatype.

I feel that you don't understand what locality is about: the property of IDs that were generated closely together in time to be close together numerically. You don't get that by "hashing" the UUID (which makes no sense anyway, since you might as well just take some truncation of the UUID). In fact the whole idea behind a hash is to destroy this property of the input data. The reason the numerical locality matters is that it is much more efficient to in-sequence-insert several numbers that are close together into an index than to in-sequence-insert the identical amount of randomly distributed numbers.

The GP was talking about the first graph here: https://www.2ndquadrant.com/en/blog/on-the-impact-of-full-pa...

I thought you were talking about cache locality within the CPU. Real cache locality.

If you care about the insert location, why not just add a brin index on a timestamp field and use that instead of assuming that the index is sequential.

CPU cache locality is not "real cache locality", it is just another place where caches exist and locality is optimized for. Your solution wouldn't work for CPUs either, other than that you could fit more data into a cache line - but that's like saying "increase the cache size to solve this problem", which obviously can help but is not addressing the inherent issue.

How would adding a BRIN index help in any way with reducing the discussed problems, such as write amplification?

This is a bit confusing, as it mixes two things - BRIN index and index on a timestamp.

The main source of write amplification comes from updating random pages of the btree index. Imagine inserting 10 random UUID values into a large index - it's pretty likely those will go into 10 different leaf pages. And every first update of a page after a checkpoint (which typically happens every 30 minutes or so), we have to write a FPI (i.e. the whole 8kB page) to WAL. So because btrees are based on ordering, random values end up on random leaf pages, causing write amplification.

If you have BRIN index on UUID column, this does not happen, because the index is not based on ordering but location in the table. If the 10 rows get appended to the same table page, that'll be just 1 write, with one FPI.

This is why BRIN does not have the write amplification issue. But it's also a bit pointless, because BRIN on random data is pretty useless for querying (Well, at least the minmax indexes, are. Let's ignore BRIN bloom indexes here.)

If you create BRIN on timestamp, that's not going to have write amplification problem, and it'll be good for querying. The thing is - BTREE would not have write amplification problem either, because the timestamps are going to be sequential (hence no updates to random leaf pages).

Right, you just need to avoid having the btree index on the UUID field. Similarly, don't pose queries to sort by the randomized UUID field either. This is where the often maligned hash index type could be useful, to allow lookup of individual rows by UUID without all the expense of a btree index maintenance. Use other fields for ordering that have better write locality.

What this means in practice, of course, is that you shouldn't expect to do application driven pagination with UUID keys either. You would need to expose some other boundary marker with a total order that works well with btrees. And this could bring you back to "leaking" predictable key material that you were trying to hide by adopting UUIDs...

Hashes evenly distribute information regardless of the input. You will never have locality unless your dataset is so small that all of it can fit into a single cache line.

Your solution just doesn't really make any sense. The time ordered UUID suggested in the blog post makes way more sense.

Reply on news.ycombinator.com