Hacker News Reader

Comment by erikerikson | Hacker News Reader

Super simple to detect and try again.

A collision is simple to detect but it requires you to actually check, which is expensive at scale. The entire point of UUIDv4 is that you don't have to check for collisions because it should never happen. But if you don't check and it does happen you are in UB territory which is generally very bad.

A risk of collision before it happens is non-trivial to detect but this is really what you'd want.

Only expensive if you have unsorted keys or lack an index. Neither of which are unscalable.

You must have missed the “at scale” part. There is nothing inexpensive about extra network hops, cache misses, and page faults implied by your solution. Indexing at scale is almost always lossy for performance reasons. The location where you insert a new record is frequently not the same location as where you have to search for an existing record.

It is resource amplification all the way down. In a lot of systems that index these keys the cost of that check is several times that of doing a blind insert.

No I didn't miss it.

DynamoDb works fine, using CQRS if necessary.

literally the whole point of randomly generating UUIDs is that you don't need to check for collision. that's what the "U"s are for. that is the abstraction that is supposedly being provided. "using <insert Amazon AWS Certification Test Answer #7>" is not in any way a "scalable solution" for that with no other context. nor is just throwing out <random Martin Fowler concept #27>. the whole point is that it is a global (well, per name, "universal") abstraction that can, in practice, have holes that make it so you can't use it "universal"-ly.

I totally appreciate what you are complaining about. It's always been part of the documentation for a UUID. Having had Martin Fowler as a colleague and meeting with him weekly for a bit, I'd expect him to nod along with what I've written. It's standard knowledge and part of the technical corpus. As is actually distributed unique ID generation which is also not hard.

well it’s especially not hard when you turn it into a completely different problem

AKA centralising a decentralised identifier generator?

There are better approaches like pre -avoiding collisions but generating tends to be more expensive than checking.

UUID is used where checking is difficult, think distributed devices offline at a plantation. How could checking be easier in that case? It would require infrastructure that doesn't exist. There are many other cases where it's easier to handle collisions.

Agreed. While it's an uncommon scenario it is a significant case and UUID shouldn't be used in that case because as you write checking doesn't work well. Better to use an alternative such as coordinate/reserve monotonic producer IDs that get paired with monotonic, per-producer monotonic sequence IDs to produce guaranteed unique, well-ordered IDs.

[edit: in IOT it's common to issue x509 certificates as a type of authentication which could be used instead of using producer IDs. Solutions always have to be paired to use case.]

In what world is generating a UUID more expensive than checking for duplicates? at any scale?

Walk me through that please

Yeah, that was a little sloppy but it's generating is more expensive than not generating. In more words, generating an id and validating uniqueness is more expensive than only validating uniqueness.

   Wild edge case. Curious if they ever found the root cause.

Welcome to Hacker News.

I suspect you meant to reply to a different comment. Regardless the most plausible speculation I've read here is that the RNG used to generate the UUID is low quality.

exactly lmao. that is exactly what is being presented as "scalable <full stop>". sigh.

No one has yet defined the scale but almost all of the real world scenarios people are actually encountering would be handled by either of the offered solutions.

as would be a million things that don’t involve using uuids at all. it literally means nothing to say “this can be handled by centralizing your uuid store”… if you do that, you DON’T NEED UUIDS. THAT IS THE POINT!

You seem to be insisting that UUID meets the promise of it's name, despite the long standing documentation to the contrary. Is that your intention?

In this specific case. In the case of trace IDs (an example of which is [1]) where the equivalent of UUIDs are explicitly used to avoid coordination, it’s hard to imagine how you’d reliably detect and retry.

[1] https://news.ycombinator.com/item?id=48033853

A lot of databases have a uniqueness constraint that is basically a register level compare and replace. Others have a if_not_exists which is nearly the same. If you're not targeting a serious throughput use case, it's enough. If you are then there are lots of solutions/alternatives that completely avoid coordination. On the other hand, maybe tracing protocols are robust to out of order delivery. If that won't do them sequence numbers tied to monotonic sequence IDs should be plenty. If not then I'd need very serious conversations to be convinced you're not wasting everyone's time

Reply on news.ycombinator.com