Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implications. (e.g. knowing when session tokens or accounts are created).
128 bits -> 128 bits
Alternatively, just say it's a random string ID and not an UUID.
[1] https://en.wikipedia.org/wiki/Format-preserving_encryption
If there's any way for the client to influence the input, it may be prone to DoS attacks: By my calculations, with a million random attempts, you would expect to find a cycle of length at least 435, which is over 13x the average. (Mind you, multiplying the number of attempts by 10 only adds about 72.5 to the expected cycle length, and probably no one has the patience to try more than 100 billion or so attempts.)
In any case, as I mentioned in a sibling comment, with 3 AES encryptions one can construct a 122-bit balanced Feistel cipher with a constant amount of work.
Breaking the 122 bits into two 61-bit halves and using AES as the round function for a Feistel cipher gives you a constant 3-encryption latency instead of the expected 64-encryption average latency of the cycle-walking format preserving encryption.
Alternatively, use AES in VIL mode ( https://cseweb.ucsd.edu/~mihir/papers/lpe.pdf ).
I’m afraid you won’t be able to ever rotate that key, would you? Since it’s result is externally used as an identifier, you would have to rotate the external identifiers, too.
You could rotate encryption keys whenever you want for new external id calculation, so that older external ids won't change (as they are external, they need to stay immutable).
Why not just hash it with pretty much any hash function?
If you didn’t care about index clustering then just use UUIDv4
If AES-128 is an acceptable external UUID (and likely an acceptable internal one), then you might as well just stick with a faster RNG.
The whole point here would be that you can expose something opaque externally but benefit from well behaved index keys internally.
Internal sorts and index on write with ULID/UUIDv7 which reveal potentially sensitive timestamp information, and so when that’s not appropriate a separate column can be used for…
External opaque identifiers which are UUIDv4, and if indexing speeds become an issue, can be switched to deferred indexing…
It’s a good balance and everything I’ve ever used supports this (I only needed a pretty strong representation wrapper for ULIDs since PG doesn’t validate UUID bit field structures so i can just miss-use UUID columns)
It looks like Snowflake has the same information exposure issues that using UUIDv7 or ULID for a publicly visible identifier.
Purely random identifiers are the bane of DB indices. The internal structure is sequential-ish and therefore indexes well.
Random identifiers spread work evenly between shards of something sharded by keyspace. They also don't get 'hotspotting' on more recent records - recent records frequently get more than their fair share of updates and changes, and database query planners have no knowledge of that.
Anything with a clustering index - MySQL (with InnoDB), SQL Server - will absolutely hate the page splits from a random PK.
Postgres also doesn’t like it that much, for a variety of reasons. Tuples are in a heap, but the PK is still a B+tree (ish), so it still suffers from splits. They also use half the default page size as MySQL, AND their MVCC implementation doesn’t lend itself to easy updates without rewriting the entire page.
Go run benchmarks with some decent scale (tens of millions of rows or more) on any DB you’d like between UUIDv4 and UUIDv7 as the PK, and see how it goes.
For tables this is not such a problem because of reasonably good locality (rows inserted close in time will end up close in the file too), but for indexes that's very difference. In Postgres this is particularly painful for writes, because of the write amplification.
None of this really breaks sharding (IDs are still unique and can be hashed), so no new hotspots. It can't fix the "many rows with the same ID" issue, but neither can any other ID.
For large distributed systems like Spanner you want to avoid a hotspot on a single node as that limits throughout.
However for a single-node system like PostgreSQL you want hot spots because they are cache friendly. (Locks aside)
Basically you want a hotspot that is as hot as a single node can manage, but no hotter. Even for things like Spanner you still want good cache hits (which is why they support table interleaving and other methods of clustering related data) but in general avoiding overloading single nodes is more important (as it hurts latency and efficiency but doesn't limit throughput like a hotspot would).
Might as well just use randomly generated external keys and have one table if you were doing that.
So, one key per all tokens.
Therefore there doesn't seem to be much downside in this specific case.
It feels like the right solution here is to pick between: use UUIDv7 and treat the account creation time as public information, or use an identifier scheme which doesn't contain the account creation time.
I mean it's certainly an improvement over telling everyone the millisecond the account was created, but if account creation times are to be considered non-public info, I would probably just not include any version of it in public facing user IDs. And if you do consider approximate account creation times to be public (such as HN, where anyone can see that my account was created July 1, 2015), then adding some fuzz to the timestamp seems to be a good way to avoid certain cryptographic issues.
https://app.example.org/file/1:abcdef12345
-> decrypt abcdef12345 with key "1" to yield UUIDv7 key of file (no matter what the latest key is)I think there is no good alternative to random external identifiers.
It is true that now your encryption key is now very long lived and effectively part of your public interface, but depending on your situation that could be an acceptable tradeoff, and there are quite a few pragmatic reasons why that might be true as has been described by other comments.
Edit: you can even do 64bit snowflakes internally to 128bit AES encrypted externally, doesn’t have to be 128-128 obvs
No need to encrypt, just store the external key in a table. Not that you're likely to change algorithms.
To that end, I think it's neat to be able to improve indexing on UUIDs, but it's not a security solution.
It may not be technically security, but e.g. knowing your competitor just added N products to their shop, might be a security issue for the business.
The end user should know no details about your internal key space.
We model our domain(s) using DDD, and often "The ID" really is best left a thing with meaning. Customer-id, Bank-account-number, invoice-number, email, etc. At least within the domain, it is. The business (and laws etc) already ensure there can only ever be one invoice with this number. Its terribly counterproductive to have two ID's for something. "Hey, can you have a look at invoice 20230233, because it seems the VAT was applied wrong. Hmm, do you have the UUID for that invoice and DM me that? You know, the long one with the hyphens".
I guess there isn't a one-size-fits all solution and that "it depends" very much on what e.g. "public" means.
It hardens, completes and complements other measures.
Examples of every day security using obscurity: every password and encryption key
EDIT: Thanks for the replies.
Ignore above!
Obscurity is the low bit of security. But when it’s convenient, it still helps.
Security by obscurity would be hiding your house key under a doormat for your friend to find - depending on the culture you live in you may be more or less safe but it is not security (just like hosting your ssh server on port 9384 will repel 99% of attackers but is not a security measure).
They are completely dumb. I haven't kept record, but I have the feeling that some IPs in my fail2ban list are practically in there for month or even years now.
I assume they are just sweeping the whole IPv4 range? No state, no cache. Either they successfully attack a host or they go to the next IP. Repeat 2^32 times, start again.
I'm not sure where I wanted to go with this comment. Is it _that cheap_ to constantly sweep the IPv4 range or is it _that profitable_ to do it once you have a successful attack?
You're getting that wrong: Everything else being equal, the more obscure system will always be the safer one. It's just that obscurity can easily be lost, so your system should, if in any way possible, still be secure even if fully known. In the end, however, no system is 100% secure, but more obscurity will make it harder to find the inevitably existing issues.
For a personal anecdote, I used to work in a small webshop and our software was horrible, to the point where minimal effort would have been able to compromise our servers, which were running software roughly as old as I was at the time (I want to note that I worked on improving the situation while I was there). Still, the only time we had a problem was when we took over a Joomla-hosted site, as we were small enough to not get any individual attention and your off-the-shelf WordPress or Joomla-scripts did not work on our home-brewed software.
In the end, I fully agree that security by obscurity is a weak concept and the usual wisdom of not relying on it is completely correct. Still, it's important to acknowledge that obscurity can and does help security and bring actual reasons on why you should not rely on it. Just saying "it's obviously bad" leads to an easily refuted argument and will not convince some developers, leading to worse software overall.
Not doing that is how small companies seem to get away with terrible security holes for a long time, until suddenly they don't. I've seen too many cases of companies in a position where they built a small, insecure service that's now getting shared more widely than envisioned, who don't want to spend the money to make it right, because no one has compromised it yet (that they know of), and what are the chances of someone stumbling across it - where even pointing out that it's an attack vector can earn you trouble.
This was used in the war to estimate the number of German tanks based on the sequential IDs
https://en.wikipedia.org/wiki/German_tank_problem
So just for business intelligence you don't want to leak your IDs.
Or from a list of most recently added customers/users, you can figure out the rate of signups.
Revealing timestamps is bad because it can reveal way too much information about the health of your business that you prefer to keep private, if a sequential list of ID's ever gets exposed (which is hard to prevent).
They're not monotonic.
My second job was for a company that provided internet enabled phone conferencing solutions (this was years before VoIP became widespread).
The customer ids were sequential. Couple that with an outright idiotic security flaw (the login process set the customer ID in a cookie and the app trusted it on ever subsequent request. Just the ID. Nothing else), I was able to iterate over all the customer ids and hand over a complete list of users to my boss to illustrate the problem, starting with a list of the accounts of the complete upper management.
They could have been used to spin up huge numbers of 30-person long distance conference calls at high costs (this company was building out nodes with 20,000 line pstn switches before they had customers... it was crazy, and they failed but would've failed far faster if that had been abused and they were on the hook for costs from their carriers)
Trusting that cookie was still stupid, but had it been a long random key it'd at least been a bit harder to discover and exploit (their next attempt was to base64 encode it and I had to explain why that didn't help; they then finally blowfish encrypted it, but without any time component, so still subject to replay attacks... I jumped at the first opportunity I got to get out of there)
To me, it reeks of solving imaginary problems while causing new ones.
That's about the only problem I can discern.
I don’t remember the details, so maybe it was something else and not photoshop/adobe.
If you don’t let an attacker iterate your data, all they can tell is when the ID was created.
The ordering means that you can reconstruct the sequence if you have enough of them, though.
The tank problem doesn't fit when the incrementing value is time since epoch. Integers yeah, UUIDs, KSUIDS, or any other semi-ordered thing to make your database indexes less fragmented I haven't seen a real leak issue with those.
We do a lot of computer vision and in his project, each processed object is assigned a UUID and he wanted to save images to files for each one.
So we took some time to go over various timestamp formats to be embedded into the filename to make the files sort chronologically. UUIDv7 is just spot-on solving our problem. In this use case, there are no real security considerations.
Obscurity can be helpful as part of defence in depth, to reduce the impact when someone does something stupid, or to make it more difficult to extract information that might be helpful as a means to attack the system from another angle.
If you're already thinking about the implications, you can likely ensure people doesn't jump to the conclusion that the IDs can be trusted just because they look complex.
They are compact, don't leak information, and make a good case why k-sortable IDs are unnecessary, or even harmful for performance.
I'm using sequential integers and created_at/updated_at timestamps for internal use, and Cuid2 IDs externally.
https://github.com/paralleldrive/cuid2#the-contenders
Some of the arguments mentioned are explained elsewhere in the README, others are assumed.
One argument standing out for me is the lack of collision-resistance for UUIDv4 which is surprising for me and I didn't spot any sources for that argument.
Another argument is the entropy source where they go about that Math.random is not reliable as a single entropy source but glimpsing at the source code, they sprinkle the CUID with Math.random data.
I am no expert in ID security, so I am not qualified to speak about the validity of their arguments, only that there's insufficient information to validate without prior knowledge about the problem domain.
Collision of UUIDV4 (which are 122 bits of entropy) are unlikely enough that it should fit most definitions of the word "impossible".
The argument listed in this library README feels like total bullshit to me, I'd avoid using it for this reason alone.
1. Collision resistance / "weak" PRGNs used to generate UUIDv4. Firstly, these are properties of the implementation, not the spec. Secondly, the source for calling the browser `Crypto.getRandomValues()` insecure is an issue that has been fixed back in 2016. I would not trust the developers of this implementation to do a better job than current browsers.
2. "Not URL or name friendly": Fair, but not very strong argument.
3. "Horizontally scalable" and "offline-capable": No argument given for why UUIDv4 does not meet these requirements, apart from point 1 above.
4. "Too fast": No argument given for why having a slower algorithm to generate random ids is more secure. Both UUIDv4 and Cuid2 use a similar number of random bits (122-124). When using a secure PRNG, both are equally difficult to guess, the SHA3 hashing doesn't add anything. You don't have to try and guess the "input" of the Cuid2 - you can just try to guess the "output" and skip the SHA3 hashing. It would be impossible to actually guess a generated ID, but UUIDv4 is just as impossible. Also no argument given for why UUIDv7 is fine but UUIDv4 is not.
I've used UUIDv4 for genering unique IDs for over 10 years now. I have run into collisions, when I hand-rolled my own implementation for J2ME with major bugs many years ago - ended up with around 20 bits of entropy instead of 122. That's not a reason to not use UUIDv4, just a reason to not implement it yourself unless you really know what you're doing.
Found them because systemd-journald isn't very happy when Boot IDs repeat and (apparently, then) stops showing earlier boots once it hits a repeating boot ID. And I wanted to see an earlier boot. Then I started logging the Boot ID in a textfile myself and it took less than 10 reboots to have duplicate Boot IDs.
Long story short, some weeks earlier, I "optimized" the Kernel config for that system and some config flags that didn't sound like something I'd need. As it turns out, an ARCH_ZYNQ target apparently also needs ARCH_VEXPRESS set. Otherwise it works absolutely fine, but with a broken RNG that you will notice weeks later.
That was a valuable "don't take down a fence until you know the reason why it was put up" lesson. Don't unset kernel config flags until you know why they are set.
Aside from breaking RNGs, I've never experienced any UUID collisions either.
If the algorithm is too fast it means you can detect when some other part of the system is having a significant impact on how the key is returned. Eg checking a database to see if a user exists and returning their key versus getting null back and generating a new key. That difference can be used to determine if a user exists. You want your key gen process to be slow enough that it's a significant part of the process, which makes timing attacks hard.
ID generation should usually only happen when creating new assets, so it should be as fast as possible.
From the article. I'd like a lot more exposition on that, since it goes against some of what used to motivate UUID use in the first place. Sequential ordering across distributed nodes isn't a fun thing to do, and even if you navigate the coding, the network agreement makes it really slow.
Do they mean "sequential enough" but some locality of the node that generated it? And I guess that prng can sometimes have some performance bottlenecks, but compared to doing locks on a single incremented integer?
Yeah, I don't really get this, lots of usual "better" "faster" etc without actual numbers to back it up or detailed algorithmic discussions.
As you basically said, wake me up if it's good enough to get into the standard vetted libraries of UUID generation.
> Cuid2 has been audited by security experts and artificial intelligence, and is considered safe to use for use-cases like secret sharing links.
I'm getting some snake oil vibes from this... There absolutely shouldn't be anything like a random ID that is 'too fast' to compute. You might need a rate limit to stay within your collision bounds, but CPU usage is a poor way to do it.
And there is currently no publicly available "artificial intelligence" that would be useful in a security audit, unless you want to call fuzzers "AI".
> One reason for using sequential keys is to avoid id fragmentation, which can require a large amount of disk space for databases with billions of records.
Disk is cheap but not free at higher tiers. But more importantly, record fragmentation means more pages (unless you take the time to do a full table lock and rewrite it, and who’s doing that?) which means more index bloat. I assure you, that adds up once you’re into the billions of records level.
> the ids will be generated in a sequential order, causing the tree to become unbalanced, which will lead to frequent rebalancing.
Given the width of B+trees used in DBs, I doubt they generally need to go more than one or at most two levels up. I’ll take the ability to rapidly follow the leaf nodes and have a good shot at sequential reads in cache from prefetch, thanks.
I don't think this is really true? These are not serially incrementing, they just indicate the time it happened. If you have an ID that you know exists, having the ability to know _when_ it was created is very rarely meaningful.
What could present more of a risk is being able to predict a large part of IDs that will be created. Even then though, you shouldn't depend on your IDs for secrecy - best to ensure the IDs are never used as protection by themselves (ie treat them like they're just a simple autoincrementing number, even if they're not)
You would be correct if the ID were an integer being serially increased. I can sign up to your website today and get an ID X and then sign up again in a week and get ID Y, I can then calculate the number of new users you've had by performing Y-X.
If this ID is a timestamp then there's no such information I can get out of it from a small sample. I sign up today and get todays timestamp, then I sign up next week and get next weeks timestamp..?
There are certainly mitigations that can be made and not all things are equally valuable as they age. (Plus many public APIs include created/modified timestamps anyway. The information is often easy to discover even when not embedded in an ID.) I don't find it a strong reason to avoid timestamp-based IDs for the threat models of that many things beyond user accounts and other things susceptible for social engineering, but it is something to keep aware of.
With UUIDv7, you are reasonably sure that there won’t be collisions (check your use-case first), and can just generate them wherever on-demand (no locks required).
I’d argue batching IDs is actually more complicated than UUIDv7 for most use-cases.
/s
Persistent IDs are a security and information risk. If that's a concern, don't persist IDs.
Here uuidv7 will just re-order that. So the content of the uuid in itself does not change.
IMO it's not the job of the identifier itself to prevent information leakage vulnerabilities though; if thee is sensitivity to this, the solution should be explicit, such as employing a secondary key derived from the UUID using a secure KDF or similar.
I parsed the EV chargers APIs where I live (using Frida in Android) and one of the fields returned the daily revenue and profit.
I've seen a project for a trading firm that inferred all kinds of traffic and revenue numbers for companies before their quarterly earnings were made public. It wasn't perfect, but knowing with a certain confidence level whether the numbers were going to be better or worse than estimate was profitable for them.
And the indexing argument isn't really compelling, is it? You lose very little by sticking to fully random UUIDs.
Pizza itself might be a bit of a joke but looking for non-operational behavioral changes is absolutely real. The Cuban missile crisis was started in part because soviets played soccer and Cubans played baseball and the presence of soccer fields helped confirm soviet presence (in enough numbers to bother making rec centers). A more advanced version might be the public Strava data leaking US base layouts and locations or Strava helping the Ukrainians kill a Russian submarine commander.
Edit to add: you could also just figure out where your target orders pizza from and pay one of the dudes working there to tell you when there’s a spike in deliveries to your target.
It's a Henri Kissinger's quote, but it's not accurate: Cubans do in fact play football. Also this quote wasn't from the 1962 Cuban missile crisis, but to another event in 1970. That being said, it is true that the US intelligence got warned by the construction of football fields (or maybe even more so by the lack of baseball grounds).
https://www.cracked.com/article_31335_that-time-soccer-field...
iswydt
If you have 74 bits of entropy, the birthday paradox says that after 2^37 keys you will hit a 50% risk of a key collision. Whether that is going to be a problem depends on the use case, and on the quality of your RNG.
So what if your microblogging platform's tweet uuid happens to collide with my cocktail recipe generator's ingredient uuid? The likelihood that any one device will ever be running our apps at the same time and trying to read data from one into the other is even smaller than the 1-in-380000000000000000000000000000000000000 (uuidv4) probability of having a collision in the first place.
Applications use uuids to avoid colliding with themselves. Uuids exist so that web-apps can create objects client-side without waiting for a database CREATE, applications can be built with multi-node dbms, operating systems can name hardware compenents.