Postgres sequences can skip 32 unexpectedly
incident.io
incident.io
I consider ID numbers somewhat opaque, like GUIDs but maybe not that opaque. Fretting about gaps in ID numbers can cause hair loss. This is just one of many ways gaps can happen.
It is just an artifact of "sequences", the database object in Postgres that autogenerates the "next" ID number for a column --- automatically set up if you declare a column of type "serial", but that's all a serial column is. Serial just means: make the column of type integer, and create a sequence for it, and set the column's default value to nextval(sequence).
You can mitigate gaps somewhat, like if you're testing and retesting a bunch of inserts, rolling back each time. Well, after a few tests, the next number in the sequence is far beyond the last number in the table. So you can call another sequence function, setval, to reset it. Something like:
select setval('sequence_name', (select max(id) from table));
Either way, it's better done in a column that isn't the primary key.
Postgres docs-- https://www.postgresql.org/docs/13/sql-createsequence.html > Because nextval and setval calls are never rolled back, sequence objects cannot be used if “gapless” assignment of sequence numbers is needed. It is possible to build gapless assignment by using exclusive locking of a table containing a counter; but this solution is much more expensive than sequence objects, especially if many transactions need sequence numbers concurrently.
Generating the sequence in application logic can make it challenging to guarantee we don't duplicate numbers if two requests come in at once (you don't want locks/global synchronous state in the application if you can avoid it), so having the database generate them is a good fit.
Normally I'd agree with you, but in this case we're explicitly dealing with external IDs that we want to be incrementing.
Our internal IDs and API IDs actually follow a totally different approach, and hold no meaning other than as a reference :)
Google Analytics either doesn't do this or got really unlucky. But some day I had to troubleshoot an issue where the data for a totally unrelated website ended up in someone's GA data set. Not just the usual spam, but millions of visits to the wrong website which had an account ID very similar to the client's.
The entire article shows that the author has a very good grasp as much of the technical side of development as of the business side. And the conclusion makes a lot of sense.
Sequences are a technical solution for a technical problem, how to uniquely identify new elements in a fast and reliable manner. And it is optimized for such use case. But, in the real world, users have expectations and when their mental model conflicts with the inner working of an application you end creating confusing and lack of trust. Depending on the situation, to educate the user can be the path to follow, but here it seems reasonable to just adapt the inner workings to the mental model of the users. It's faster and scales easier as the number of customers increases.
We were very open, and our customers in this case were really understanding - just one of the reasons we love working with them!
Be careful about treating sequences as monotonic. For transactions in progress at the same time, the order of sequence values might not be consistent with the order of transaction commits and the logical order of serializable transactions.
One example where that could cause problems is if you filter a change stream using the last seen id. For such an approach an out-of-order id would lead to missed events.
Luckily our _internal_ IDs don't rely on sequences at all, and for ordering I'd always use a field specifically for that purpose like `created_at` timestamps etc (vs inferring ordering from IDs). Best if IDs just remain references!
This could have been another interesting bug though, in a way glad we hit this one instead and that it's now fixed so we don't hit this one in future!
1. If you create it in the application, clocks need to be synced well enough between all application servers. If you create it in the database, this shouldn't be an issue.
2. The transaction completes some time after the the timestamp was created, and that time can vary between concurrent transactions. This is a fundamental problem.
The MAX + 1 approach should guarantee strict monotonicity, but might lead to scalability issues for highly contented counters.
Customer #C1010 may have 2 locations #L1899 and #L8443 and many invoices #IN1940 and #IN2399 for example.
When we first built the system, I considered using native Postgres sequences to track these, but decided against them because of how they are affected during a rollback. In our system, each account has a record in a table that controls the next value of the sequence.
We have an event in our ORM to automatically generate the next sequence value as part of the transaction so if the transaction is rolled back, the next sequence value is as well. Sure, it requires locking the sequence record but it's a very small table and generating a sequence is quick. We wrapped everything up in a stored procedure named generate_sequence() which returns the next value of the sequence and increments it. It's scaled to millions of records quite well without issue.
In our system, by default, all objects start at 1000. If a new account is created, and they want to increase a sequence to some value (say they already have 5000 invoices in QuickBooks and they want to start all new invoices at 10000 so they know every invoice #IN10000 and higher was created in our system), we have a simple interface that one of our support staff can go to arbitrarily increase the next value.
I assume that means you also have to only allow one TX to be in-flight at a time adding a new record whose ID is generated from a given sequence?
It's quite a natural optimisation.
with u as (
update organizations
set last_external_id = last_external_id + 1
where id = 123
returning last_external_id
)
insert into incidents (organization_id, external_id, title)
select 123, last_external_id, 'Blah blah blah'
from u;
(so performance wise, a sequence is still better due to less locking, but if you can't have holes it seems fine)
I sympathize with customers who don't expect gaps of 20 (Oracle) or 32 in ID sequences. But the relational model is one of unordered tuples, isn't it?
The only thing you can say for sure about a sequence is the next number will be greater, which is not good enough for many kinds of identifiers.
In your opinion. There are advantages of sharding tables by tenant boundaries. Data isolation and query speed to name a couple.
Aren't there a zillion ways to compromise things if you know that some field is a sequence?
It has never been a problem for me and my applications. Just because you know a record exists, doesn't mean you can see it. For example, if you are authorized to view https://www.example.com/records/100, and you decide to try https://www.example.com/records/101, then the code will check to see if you're authorized to see record 101. If not, then you will get Unauthorized.
I suppose there are situations where it's a problem if someone finds out that record 101 even exists, but not in any of my apps.
Well, it's things like say, an invoice number.
I can buy something from you. And then 7 days later I buy something else from you.
If the invoice numbers are in sequence, I just gained quite a bit of information about how fast you are selling things.
That's the kind of information leak that sequences can create.
[edit] It's even EU-wide, see article 226(2) of directive 2006/112/EC:
> a sequential number, based on one or more series, which uniquely identifies the invoice;
Today's first invoice could be 2021-07-16-001, the second one 2021-07-16-002, etc.
If you really don't want people to be able to guess your invoice volume from numbers alone, there are various ways to do that while still being compliant to EU laws.
The Italian authorities seem to see this the same way - https://vatdesk.eu/en/eu-vat-news/italy-mandatory-mentions-o...
(Annoyingly the directive doesn't give a definition for "a sequential number" itself.)
I do agree that (as is so often the case) the directive is not specific enough to determine whether the Italian interpretation is correct or not, though I can provide some context from the German side as provided by the ministry of finance: "Eine lückenlose Abfolge der ausgestellten Rechnungsnummern ist nicht zwingend"[1] (~ "it is not required for the invoice numbers to be gapless"), which directly contradicts the Italians as far as I understand it.
[1] https://www.bundesfinanzministerium.de/Content/DE/Downloads/... - p. 522, 14.5 (10) 4
But you can't determine if there are any gaps which is what the Italian ruling seemed to be concerned with (as best I could follow the Google translation which was pretty bad) and the Germans aren't bothered about.
I think if it were me, I'd ere on the side of caution and have them be sequential (ordered, gapless).
Perhaps it's a bad idea in this one specific case but not in general.
Using random id's is a nice and low effort additional layer of security there.
No:
- sequences are very common. I recommend using them on every table for mgmt. and internal efficiency reasons.
For example, with Innodb, if you don't have a numeric id as a PK, it will assign an invisible one for internal use anyway.
Most third-party tools won't allow you to manage tables without numeric PK's.
- in most large applications, most sequence ID's are only used internally
- for public display (your concern) uuids or random numbers are possible
Source: DBA.
> with Innodb, if you don't have a numeric id as a PK, it will assign an invisible one for internal use anyway.
There's no requirement that your PK be numeric with InnoDB. A monotonically increasing numeric ID will have the best performance, yes. But even a small-ish varchar PK may perform better than relying on InnoDB's internal invisible one, depending on the workload.
InnoDB will only use an invisible numeric PK if you have no explicit PK defined, and you either have no UNIQUE KEYs at all either, or all of your UNIQUE KEYs have nullable columns. The column type of your PK is irrelevant though.
The invisible numeric PK is terrible because it uses a system-wide lock (or at least it did prior to 8.0, not sure if this has been fixed). So if you're inserting at any real volume to multiple tables like this, performance suffers badly. Worse still, the lock it uses is the dict_sys mutex, which other code paths (e.g. DROP TABLE) also hit.
It is usually not something to worry about, but in some cases you want to avoid leaking that info.
The Allies estimated how many tanks Germany was producing based on the serial numbers, and this gave a closer result compared to other intelligence sources.
I don't think they're always a bad idea, but when talking about external IDs many folks would agree with you, in a few ways, actually.
Relying on exposed primary IDs being ordered, and/or exposing them, is often a bit of a can of worms.
A common case is that you leak information about your company because people can see how quickly you're growing. I've seen this in a few products I use and it's always interesting when you take an action a few days apart and can see how much volume they're doing.
TIL: German Tank Problem, thanks @teddyh! I first came across it with a good story about someone buying Donuts/Coffee in a shop and using the receipt numbers to estimate yesterday's sales, can't find the link, though :(
In this instance it's not our primary ID field - those are internal, and are long and random. They holds no meaning other than being a reference, so aren't used for sorting or exposed to the customer, and even if it was, it wouldn't mean anything or confer any information.
The IDs I refer to the in the article are more like external references. References we _explicitly_ want to increment by 1 each time.
For those referencing invoices as a parallel, it's a good equivalent. I'm not familiar with the details in the comments below, but if I created two invoices and they were referenced #1 and #33, I'd be quite confused (which is a version of what our customers felt/experienced here).
IMO It's also often a good idea to use different external IDs to your internal ones in APIs too, and ideally have no meaning attached to them, either. That way users don't do things like assume "record 99" was created before "record 100", and you can also move data around and migrate things, so long as you honour those external references (i.e., you can change the type/format of your internal IDs at will).