Hacker News Reader

Comment by x0x0 | Hacker News Reader

> me wonder about their redundancy and recovery architecture

You can pretty much guarantee it's untested and unlikely to work, when someone doesn't understand how drive arrays work, or test expanding drive arrays before doing it in prod...

View on HN

They messed up and either ran out of disk space or had an array fail. Everything in that statement reads like they are trying to frame a critical production failure as a glitch that occurred during maintenance. This wasn’t about a routine storage expansion. It was an emergency caused by some operational screwup and the only way to fix it is to rebuild the array with new disks.

I had a raid6 array going almost bad when 2 HDDs called it a day at the same time years ago. Had to send the entire office of 30 people to unscheduled holiday for two days before the array was rebuilt with new drives. Third drive failing during the rebuild would likely result in a week off as the storage would need a full rebuild.

I can imagine something similar happening here, however there should be a fail over storage setup in case they're selling a service.

In a storage business no less. They literally had one job: to know how to resize arrays, so the rest of us don't have to.

They messed up here in some way, but assuming you know what events and complaining about their skills is just making stuff up at this point. Unless you can tell us exactly what happened, maybe hold off with that kind of criticism.

> However, we did not anticipate that the expansion process would require the server storage to be temporarily offline.

One, it is obviously some flavor of raid rebalancing, there’s not a different kind of thing it could be.

Two, they’re admitting they didn’t expect this, that’s not possible if it’s a tested procedure.

And three, they're admitting they don't back anything up (backup defined as not just slinging bytes somewhere but the availability to restore those bytes and bring a service back up on a timeline that is certainly well lower than 3 days. With regular testing to understand the timeline and ensure everything actually works.)

ps -- it's not about being bashing these folks. But these type of multiple operational incompetencies strongly call into question their ability to safeguard your bytes and produce them when you need them. It's kinda the whole point of the thing.

It's a backup company. I can understand that they don't back up the backups.

It’s a backup company so I would expect the store 1 copy plus the minimum redundancy to make drive failures a predictable amount of unlikely to cause data loss. More than that would be additional features.

If there's one thing you'd expect from "a backup company", it'd be that they're really good at doing backups. :)

That does include their own of course.

Possible, but that's still guessing. Also possible: extra load while adding storage caused multiple drive failures, so suddenly they're without spares and want to finish the migration as soon as possible, so they stopped the customer traffic.

"did not anticipate" can be interpreted in different ways too. It may be "didn't know it could happen", or just "estimated it to be so unlikely that emergency downtime was acceptable as a response".

I've seen outside speculations as an inside person before and they're often very likely to be BS. Let's stick to facts and skip the "obviously" and "not possible" until we learn more. What's happening right now may be their prepared and tested procedure. (Or it may not)

By the way, is this a plausible scenario?

(1) Add more drives, start online rebuild

(2) Have one drive fail/URE mid-rebuild

(3) Be forced to switch to offline rebuild because the array was in a strange not-entirely-consistent state even before the failure, even though without the new drives you’d be in a routine online-rebuild situation

If so, possibility (2) could in principle not be detectable in staging, as it could depend on the drive’s age, and having to reread everything—as a rebuild does—is a rather abnormal load. The failure could even predate (1), if it happened to data nobody’d looked at for a long time.

This is less a question about these guys and more in general—is a RAID/ZFS/etc. array/pool/etc. in a vulnerable state after you’ve expanded it?

> and having to reread everything—as a rebuild does—is a rather abnormal load.

Far from being an abnormal load, it should be a cron job. Backups need to be verified. Data at rest needs to be periodically checked for bitrot. Reading everything on the array and verifying parity and any checksums is important preventive maintenance and an early warning system.

That a decision needs to be made under uncertainty doesn’t relieve one from the necessity of making the decision. That implies estimating an uncertain state of the world and drawing inferences from that, and whether those are framed as “criticism” is irrelevant.

In situations where you could plausibly get more data, it’s usually OK to admonish people to just go and get some, because whatever the difficulty is doing that, judging the current degree of uncertaintly is probably even harder. In situations where you couldn’t, though (trade or state secrets), there’s nothing to do but to use what data you do have and point out your uncertainty carefully.

(This is a very general argument because what you’ve said is a fully general counterargument to any criticism of anyone from anybody but an insider. As for this particular case, GP’s allegation of lacking ops knowledge seems to be on the charitable side to me: an uncharitable one, as stated elsethread, would be that they had one drive too many fail on them, whether due to incompetence or misfortune, and are forced to do an offline rebuild while choosing to lie about it. It’s the lie part that I’d have the greatest problem with, if it’s actually there.)

This is why I ultimately trust my personal data to Apple (and to a lesser and smaller extent, Google). I don't expect either to exactly have my best interest at heart - but I do expect both of them to have basic ops competency.

If you have multiple backups, you don't have to trust anyone.

except yourself !

Actually if I must entrust my data to companies like these where reliability and competence is of utmost importance - I’d rather trust Google and Facebook (much much before Apple).

If we are talking about privacy (or rather privacy theatre) then yeah Apple, why not.

I pulled down my Google Drive data some time ago with rclone and a bunch of the older files were corrupted.

I wish Apple made a Time Capsule that requires iCloud 2TB Subscription to work.

They could have priced it at $299 to include the initial 12 months iCloud subscription.

Sorry, can you expand on that? Why require the sub?

Old Time Capsule isn't really a backup. You either need two HDD with BTRFS or ZFS set up as minimum. And then You will need an offsite backup option. iCloud , Google storage etc.

If you only go via the iCloud route, you have zero local copy of your data. If you only only go via local, it isn't safe enough option. So ending up with a combination of the two.

And Apple really wanted services revenue, if it was only just a Time Capsule I doubt Apple would ever make it again.

Large corporations have the technical skills in place, but organizationally they can lock your account away leaving you with no access and no real avenue to evict your data. There are many posts on HN and on the Internet where corporate support had no procedure to give back access to data when they blocked the account. Account bans are sometimes for unclear or out of control reasons (e.g. someone stole your credit card and then used that card with a another account, so the corporate banned all accounts indefinitely). In the case of a smaller organization, you have a better chance that if they have to say goodbye to you, they will do it without unnecessary harm to you.

That would also be a major security concern: people gaining access to accounts by social engineering .

Well, you do at least encrypt your backups yourself, right?

That doesn’t prevent the deletion of backups and snapshots, encrypting the encrypted backups for a ransom, and using your account for distribution of the illegal content or malware.

Sure, but lets try and be a little more realistic and weigh the scenarios accordingly.

Yeah, I wouldn't trust their process at all, if they'd rather have the server down for 1 week instead of aborting and performing a rollback to the last known good state.

I suspect they can't. Abort and rollback, that is. Evidence: it's not rational to eat a week of downtime if you have an alternative.

edit: Which tells you something about their backups, and it ain't great. Or they'd restore onto another box.

Reply on news.ycombinator.com