Subject: Tarsnap outage 2023-07-02
From: Colin Percival
Date: Sun, 2 Jul 2023 10:13:26 -0700
The important information first: All data stored on Tarsnap is safe, but you
can't access it right now. I'm working to bring it back online.
The detailed story:
At approximately 2023-07-02 13:07 UTC the main Tarsnap server suffered a
"failed system status check" in Amazon Web Services vernacular. In plain
language, something broke -- either the server hardware, the power, or its
connection to the rest of the EC2 network (including network-attached disks).
At approximately 2023-07-02 13:52 UTC the server rebooted (probably running
on new hardware, considering that it's a virtual machine), but had filesystem
corruption affecting cached metadata. This cached metadata essentially tells
the Tarsnap server where to find user data in Amazon S3; without it, the
Tarsnap server cannot service requests. I *think* that the metadata present
is accurate as far as it goes, but it definitely lacks information about
archives stored in the last few seconds before the outage started.
While I could probably "replay" the most recently stored data from the backing
store in S3, the nature of filesystem corruption means that I can't guarantee
that this would work; since Tarsnap is a backup service I have to prioritize
safety, and so I've started the process of reconstructing the Tarsnap server
state from the "known good" data stored in Amazon S3.
If anyone needs data urgently, I can bring the Tarsnap server up in read-only
mode, subject to the caveat that the most recently stored data may not be
available. Please contact me if this would be useful to you.
It's too early in the state reconstruction process to provide an accurate ETA;
my best guess is somewhere between 12 and 24 hours.
As always, feel free to email me if you have any questions.
Sorry about the headaches,