The dangers of resetting everything at once
rachelbythebay.com
rachelbythebay.com
The 49.7 day bug: https://support.microsoft.com/en-us/kb/216641
Apparently it's caused by uptime being stored in a 32-bit integer which simply overflows at that point in time
https://www.ibm.com/developerworks/community/blogs/anthonyv/...
A couple of years ago we had a total power failure of our major data center and central office, including the backup gen so literally everything failed and an entire province of Canada (and most of two more) were without any telco, data, wireless.
It was eye opening and extremely scary to see what happens when everything is turned off at once and you need to bring stuff back up. Some of the equipment had never been turned off, and certainly not all at once.
During the next weeks we encountered many, many catch-22 scenarios like "Before we turn on System A, it needs system B to be online. B needs C, C needs A." Oh.
For reference of how severe this is, paramedics were posting up at major intersections in towns and communities in case people needed help - and the radio stations in half of Canada were telling people what to do. When even 911 doesn't work you have problems.
It took out everything in the Yukon, and tons of stuff in NWT, Nunavut and Northern BC and Alberta.
In my experience, a telco is the definition of "too conservative".
I have some production code that uses ints as serial numbers with a factory method that just hands out the next one in sequence, however, I take care to ensure that there will be no negative consequences if they roll over (they exist only in debug builds).
The problem is that the existence of that one issue demonstrates a serious failure in the tools and development processes and quality checks used to create the control software. There could therefore be other issues in the code that haven't been found yet that could cause similarly catastrophic failures.
Failing to reliably identify this kind of arithmetic error is a huge deal when you're working with high-reliability software. It simply shouldn't happen, and there should have been several different stages in the development process that would have prevented it.
So, the development team should have the book thrown at them until someone figures out exactly how it did happen, closes every related tool and process loophole so it can never happen again, and audits all potentially affected code to make sure the same combination of loopholes hasn't allowed any similar mistakes into production elsewhere in the system.
Then it's at their discretion what do (for example, decide when to reboot).
"Everybody" knows that no good Web 3.0 company has an ops team. We're too expensive and too easily replaced with a handful of shell scripts or can just be fobbed off as another responsibility of the devs who wrote the code. (After all, if the code was perfect, there wouldn't need to be an ops team, right?) Silly rabbit, ops teams are for lumbering companies who "Just Don't Get It."
Signed,
Slightly miffed ops person who thinks he does a lot for his company but feels woefully underappreciated in the new IT.
I would hope for that, too. That's certainly how my businesses operate.
Sadly, reading sites frequented by the start-up community, including HN, taught me long ago that plenty of entrepreneurial types will feel absolutely no guilt about skipping things like security and privacy safeguards if it gets them significantly more/quicker money. They just hope that they will be able to handle any PR fall-out if it ever becomes necessary, and it's one more risk to manage, nothing more.
If something really bad happens, their back-up plan is simply to fold the business and start a new one. They'll write off the loss without much regard to the customers who had supported them or any damage that might be caused to those customers by the leakage of that sensitive information. In short, your second characterisation is all too realistic.
I think this is almost inevitable as long as the start-up culture is focussed around either having an outside shot at being the next Google/Facebook/Apple, having a realistic chance of being acquired by the current Google/Facebook/Apple within a fairly short period, or throwing it all away and starting again. By its nature, this business attracts gamblers. Lacking any meaningful penalty for not taking proper precautions, not just for the start-up but also for the founders/leadership of the start-up and their investors, the odds are more in favour of those who cut corners. Looking out for your customers can even be a direct competitive disadvantage.
To change the culture, you need to change the attitude of either the founders or the funders. The former would take something like piercing the corporate shield and making the officers of a company personally responsible for negligent data leaks, probably not just in monetary terms but also something they can't shake like barring them from being officers of any other company for some significant period afterwards (thus killing the dump-it-and-start-over strategy). The latter just needs a direct financial penalty severe enough to make cutting the corners at the risk of user data not a good bet, which in practice is probably much easier to achieve, and without the negative side effect of making honest but nervous founders more reluctant to take a risk on starting a business.
Signed,
Developer who wants to develop rather than being a part-time, under-trained, unsupported sysadmin and who finds too many stories about modern devops environments funny for all the wrong reasons
I've worked at places where we managed lots of systems, and we weren't quite organized enough to jot down when known issues would crop up far in the future and remember them. SSL certs specifically bit us once or twice.
As long as your alerting system allows custom alerts, you'll always be able to run such a check.
Not guaranteed a valid assumption. (In fact, anecdotal evidence suggests it's the exception rather than the rule, at lest for businesses below a certain size.)
I think that's too much of a generalisation. If you're talking about an established public service, that you're charging real money for, where something that actually matters will be affected by even minor downtime, sure. But if you're talking about a small team or individual, running a new service that does something simple to help someone do something else, you probably have many higher prioritises than that level of monitoring and alerting, but you might still get messed around by something like all your certs expiring overnight.
It may have produced a global phenomenon where people observe multiple expired certificates around that date; however, there is absolutely no difference to an end-user of a particular site whether the cert is expired in April or October. They are both as equally bad.
In response to heartbleed, a system admin would have gained no advantage by waiting to react and would have been exposing his/her users to MiTM attacks by waiting longer.
https://hn.algolia.com/?query=ssl%20expired&sort=byDate&pref...
I suspect the real issue is that in "emergency situations" people get the important shit done (and replace certs immediately), but don't always do the non-emergency process type stuff that'd normally get done when doing important shit by the schedule (updating reminders about cert expiry dates).
My guess is Instagram, senate.gov. gmail, and Docker (amongst others) are going to have ops people wandering around over the next few months saying "Hey, we just got the ssl cert update reminder, but someone's already renewed the cert months ago and didn't update 'the system(tm)'. What gives, people?"
A fix would be to update those new certificates proactively, spreading out the workload and new expiry times. But also marking the calendar for mass expiry day because there are bound to be some that were forgotten.
> So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar. That's the first possible date you should see a cluster of events beginning.
What? Doesn't this assume the announcement date X was when the bug was introduced into the software (in the example the Linux OS) and installed on your servers?
Let me try restating one sentence. "That's the first possible date you should see a news story or otherwise notice a cluster of events around the world as sysadmins everywhere inadvertently synchronized their systems by rebooting on day X"
> So here's the trick: any time you see an announcement on date X of something bad that happens after item Y has been up for more than Z days, calculate what X + Z is and make a note in your calendar.
I thought she was making this suggestion to the 'kneejerk' admins, not us clever admins watching for failures externally.