Comment by smcleod | Hacker News Reader

Comment by smcleod | Hacker News Reader

Their current ETA for service restoration is another 2-4 days.

Despite the warning banner the uptime for customers on us10 instance which has been unavailable for almost a week now is showing at 100%.

They are compensating affected customers by crediting 4 weeks subscription but I must say this does make me wonder about their redundancy and recovery architecture.

Opshugs to the folks at Borgbase.

--

> box-us10 offline for storage expansion

> We are currently experiencing a temporary outage on our box-us10 server due to an unplanned expansion process. We added two new hard drives to the server in an effort to enhance capacity and accommodate future growth. However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

> To prioritize the safety and integrity of your existing data, we have decided to keep the box-us10 server offline until the expansion is successfully completed. It's estimated that about a week of downtime will be needed for the expansion. We understand the inconvenience this may cause and sincerely apologize for any disruption to your services.

> Current progress: 26% (Sunday morning UTC) > Date Created: 2023-08-11 16:17:36 (3 days ago) > Last Updated: 2023-08-13 18:15:53 (15 hours ago)

> me wonder about their redundancy and recovery architecture

You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

Yeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.

I suspect they can't. Abort and rollback, that is. Evidence: it's not rational to eat a week of downtime if you have an alternative.

edit: Which tells you something about their backups, and it ain't great. Or they'd restore onto another box.

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

If you have multiple backups, you don't have to trust anyone.

except yourself !

Actually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple).

If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.

I pulled down my Google Drive data some time ago with rclone and a bunch of the older files were corrupted.

I wish Apple made a Time Capsule that requires iCloud 2TB Subscription to work.

They could have priced it at $299 to include the initial 12 months iCloud subscription.

Sorry, can you expand on that? Why require the sub?

Old Time Capsule isn't really a backup. You either need two HDD with BTRFS or ZFS set up as minimum. And then You will need an offsite backup option. iCloud , Google storage etc.

If you only go via the iCloud route, you have zero local copy of your data. If you only only go via local, it isn't safe enough option. So ending up with a combination of the two.

And Apple really wanted services revenue, if it was only just a Time Capsule I doubt Apple would ever make it again.

Large corporations have the technical skills in place, but organizationally they can lock your account away leaving you with no access and no real avenue to evict your data. There are many posts on HN and on the Internet where corporate support had no procedure to give back access to data when they blocked the account. Account bans are sometimes for unclear or out of control reasons (e.g. someone stole your credit card and then used that card with a another account, so the corporate banned all accounts indefinitely). In the case of a smaller organization, you have a better chance that if they have to say goodbye to you, they will do it without unnecessary harm to you.

That would also be a major security concern: people gaining access to accounts by social engineering .

Well, you do at least encrypt your backups yourself, right?

That doesn’t prevent the deletion of backups and snapshots, encrypting the encrypted backups for a ransom, and using your account for distribution of the illegal content or malware.

Sure, but lets try and be a little more realistic and weigh the scenarios accordingly.

They messed up and either ran out of disk space or had an array fail. Everything in that statement reads like they are trying to frame a critical production failure as a glitch that occurred during maintenance. This wasn’t about a routine storage expansion. It was an emergency caused by some operational screwup and the only way to fix it is to rebuild the array with new disks.

I had a raid6 array going almost bad when 2 HDDs called it a day at the same time years ago. Had to send the entire office of 30 people to unscheduled holiday for two days before the array was rebuilt with new drives. Third drive failing during the rebuild would likely result in a week off as the storage would need a full rebuild.

I can imagine something similar happening here, however there should be a fail over storage setup in case they're selling a service.

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be.

Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.)

ps -- it's not about being bashing these folks. But these type of multiple operational incompetencies strongly call into question their ability to safeguard your bytes and produce them when you need them. It's kinda the whole point of the thing.

It's a backup company. I can understand that they don't back up the backups.

It’s a backup company so I would expect the store 1 copy plus the minimum redundancy to make drive failures a predictable amount of unlikely to cause data loss. More than that would be additional features.

If there's one thing you'd expect from "a backup company", it'd be that they're really good at doing backups. :)

That does include their own of course.

Possible, but that's still guessing. Also possible: extra load while adding storage caused multiple drive failures, so suddenly they're without spares and want to finish the migration as soon as possible, so they stopped the customer traffic.

"did not anticipate" can be interpreted in different ways too. It may be "didn't know it could happen", or just "estimated it to be so unlikely that emergency downtime was acceptable as a response".

I've seen outside speculations as an inside person before and they're often very likely to be BS. Let's stick to facts and skip the "obviously" and "not possible" until we learn more. What's happening right now may be their prepared and tested procedure. (Or it may not)

By the way, is this a plausible scenario?

(1) Add more drives, start online rebuild

(2) Have one drive fail/URE mid-rebuild

(3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation

If so, possibility (2) could in principle not be detectable in staging, as it could depend on the drive’s age, and having to reread everything—as a rebuild does—is a rather abnormal load. The failure could even predate (1), if it happened to data nobody’d looked at for a long time.

This is less a question about these guys and more in general—is a RAID/ZFS/etc. array/pool/etc. in a vulnerable state after you’ve expanded it?

> and having to reread everything—as a rebuild does—is a rather abnormal load.

Far from being an abnormal load, it should be a cron job. Backups need to be verified. Data at rest needs to be periodically checked for bitrot. Reading everything on the array and verifying parity and any checksums is important preventive maintenance and an early warning system.

That a decision needs to be made under uncertainty doesn’t relieve one from the necessity of making the decision. That implies estimating an uncertain state of the world and drawing inferences from that, and whether those are framed as “criticism” is irrelevant.

In situations where you could plausibly get more data, it’s usually OK to admonish people to just go and get some, because whatever the difficulty is doing that, judging the current degree of uncertaintly is probably even harder. In situations where you couldn’t, though (trade or state secrets), there’s nothing to do but to use what data you do have and point out your uncertainty carefully.

(This is a very general argument because what you’ve said is a fully general counterargument to any criticism of anyone from anybody but an insider. As for this particular case, GP’s allegation of lacking ops knowledge seems to be on the charitable side to me: an uncharitable one, as stated elsethread, would be that they had one drive too many fail on them, whether due to incompetence or misfortune, and are forced to do an offline rebuild while choosing to lie about it. It’s the lie part that I’d have the greatest problem with, if it’s actually there.)

In a storage business no less. They literally had one job: to know how to resize arrays, so the rest of us don't have to.

Are they using big disks and having to resilver or re-do their RAID setup across the added drives, requiring everything on the disks to be read and re-written?

It would appear to be something like that.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category. Like not an accident or fat finger or a bug, but engineers not understanding fundamentals of how a system worked and having none of the processes in place to to dry runs in non production or any of the other things that would protect against something like this. Really questionable to trust an organization that makes a mistake like this.

It's not an org but a one man show. I'm using borgbase myself, really happy with it until now and I have to admit I'm now reconsidering this choice...

One man show has a bus factor of 1. Which is kinda bad for anything critical...

"This is embarrassing. Somebody made a big mistake of the “mistakes like this shouldn’t be made” category."

You have no idea what happened.

Protecting against "something like this" often introduces complexity and failure cascades that are much more harmful than a simple system being offline.

In fact, I would consider it a feature if uptime was deliberately sacrificed for simplicity and data integrity.

I wish Manu the very best and will consider it not a failure but a success if this array/subsystem/whatever comes through without data loss.

>unplanned expansion process

Sounds like an Elon Musk euphemism after a bad day at SpaceX.

Someone added disk to an array without understanding what would happen next.

What happens next? Shouldn't the array continue working, albeit with significantly degraded performance?

Well, in my sysadmin days we would have test/staging equipment that we'd make sure we knew what would happen when a change was made.

But I don't know the details of what really happened, so I can't make a judgement really. If I could edit my previous comment I would, to be less emphatic that someone did something "wrong".

It could have been that this particular disk array had different firmware or was faulty. All we know is that something unexpected happened.

Sorry, I didn't quite answer your question!

Yes, the expected behaviour would be for the disks to be initialized. Beyond that, it'd be a configuration setting for what to do with new disks. It could be used to extend the existing LUNs, or added as a new pair in a RAID 10 style setup, or added as new members of a different level RAID.

Then it would be up to the sysadmin to extend the LUN, or divide the newly added space among multiple LUNs.

Different arrays have different behaviour with new disk. Some only add to offline LUNs. Some do everything online with lower performance, as you said. Even some super expensive enterprise kit will be very easy to break by adding new disk.

Thanks for the insightful answer!

Reply on news.ycombinator.com