Hacker News Reader

Comment by tgtweak | Hacker News Reader

This exact thing happened to realitychecknetwork hosting about 6 years ago (now rebranded to serverstack and digitalocean).

There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.

"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.

They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.

They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.

Good cautionary tale for segregation of credentials and proper user key management.

View on HN

Looks like DO has managed to keep this detail out of the record for the most part, save some forum posts that pop up if you search "realitychecknetwork digitalocean". It would explain the odd gap in time in their Wikipedia page history:

> In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.

Yeah Ben and Moisey are both incredibly competent sysadmins and shrewd businessmen. They deserve the success they've had but they were very close to desperation after this incident. I can't blame them for omitting it from the records, the reality is likely that digital ocean and serverstack are both much more secure as a result.

Problem is at some point some person or more likely group of people has to be entrusted with the Master Keys. Using different keys for production machines and that sort of thing is all well and good but they're kept somewhere, meaning if an admin decides to screw you, there isn't a lot you can do.

Seems like the better option is keep your admins happy as much as possible.

By that logic, all employees should be kept as happy as possible because they might commit arson.

All employees should be kept as happy as possible anyway, shouldn't they? Why would anyone want unhappy employees?

The point is, it's not a trade-off between employees' job satisfaction and 'committing a crime'. These are not two sides of the same or even related issues. You don't keep employees happy because they might burn the place down if you don't and and someone's departure from a job is not a reason for them to commit a crime. Equating these things is silly.

I need to watch Office Space again...

> By that logic, all employees should be kept as happy as possible because they might commit arson.

If that's the only logic that successfully gets through to the boss, it's good logic.

Well in general, you catch more flies with honey than with vinegar. I can't help but notice that most of these companies that run into these sorts of issues also had some pretty bad stories on Glassdoor.

Not necessarily causality, but I'd say there's at least a correlation, and a good enough reason to make office life as bearable as possible.

Reply on news.ycombinator.com