My fake job in Y2K preparedness
nplusonemag.com
nplusonemag.com
Y2K was a real problem. The end-of-the-world blackouts + planes falling from the sky was sensationalism, but there were real issues and most of them got fixed. Not trying to take away from this very interesting story of corrupt cronyism, but there were serious people dealing with serious problems out there. "Remember Y2K? Nothing happened!" is a super toxic lesson to take away from a rare success where people came together and fixed something instead of firefighting disasters.
Noone is being promoted for doing good job that prevented an unanticipated disaster from happening.
For accounting, it's not a simple cost center, and is at the front line to show the numbers. They can say how much it will cost to not comply with a rule, or how much they saved by creativity or ingenuity. Being that close to the money is a tremendous advantage.
Legal is more distant, but there's a clear scale of how much is on the line. When you review a contract, it's pretty clear what’s at stake if legal work is botched.
That's my main takeaway: if you care about money, you need to be as close to it as possible. At the same skill level, dealing with user security or financial transaction security won't pay the same.
See perhaps:
> Normalcy bias, or normality bias, is a cognitive bias which leads people to disbelieve or minimize threat warnings.[1] Consequently, individuals underestimate the likelihood of a disaster, when it might affect them, and its potential adverse effects.[2] The normalcy bias causes many people to prepare inadequately for natural disasters, market crashes, and calamities caused by human error. About 80% of people reportedly display normalcy bias during a disaster.[3]
* https://en.wikipedia.org/wiki/Normalcy_bias
Also maybe:
> Optimism bias or optimistic bias is a cognitive bias that causes someone to believe that they themselves are less likely to experience a negative event. It is also known as unrealistic optimism or comparative optimism.
There's also this annoying catch-22 which may be familiar to IT staff:
1. Things go wrong: "What do we even pay you for?"
2. Things go right: "What do we even pay you for?"
How the people in charge of this stuff never noticed the cycle is beyond me.
My cynicism about Y2K comes from the fact that there were a lot of snarky articles written about how certain countries or companies were not Y2K ready but nothing bad seemed to happen to those countries either. It seems like a natural experiment was conducted and the results indicate there was no correlation between good outcomes and the work done to be Y2K ready.
I have no doubt that the armies of consultants did fix real issues but anyone working in software knows there is a never ending set of things to fix. The real issue is whether that work was necessary for the ongoing functioning of business or society.
The first "Y2K" bugs where when banks' computer systems started messing up the calculations of long date financial securities/mortgages - decades before the millennium. Closer to the time Supermarkets started junking food that had a post 1999 Best Before date. Those were company ending problems if not fixed and so got overwhelming and rapid focus.
Bad things still happened everywhere, despite all our efforts. How bad depends on your perspective.
Several people suffered a bizarre form of resurrection, which normally, Christians would be all over it and jolly excited. Pensions suddenly started paying out, tax bills became due from people long dead. If you were not a relative of one of those people it did not affect you and if you read about it, you'd have perhaps said "typical" and got on with life.
Some devices just went a bit weird and needed turning off and on again. Who cares or even noticed? Someone did but again, you did not hear about those.
I spent quite a while patching NetWare boxes and applying some very shaky patches to Windows workstations. To be honest, back then, timezone changes were more frightening than Y2K - they happen twice a year and something would always crash or go wrong.
The sheer amount of stuff that was fixed was vast and I don't think your "countries that did and did not" thought experiment is valid. Especially as it is conducted without personal experience nor much beyond a bit of "ecce, fiat" blather.
Nowadays time is so easy to deal with. Oh, do you remember when MS fucked up certificates and Feb 29 a few years back?
Possibly the catastrophic things were prioritized and fixed?
I directly witnessed a few near catastrophic failures due to Y2K at different companies, literally company killers. We kept everything (barely) running long enough to shore up and address the failures without anyone noticing, partly because we had prepared to operate in the face of those failures since we knew there was no way to fix them beforehand. It was a tremendous decentralized propaganda coup. No one wanted to be the company that failed as a result, the potential liability alone was massive.
The idea that what was averted was minor is a pretty naive take. I was genuinely surprised that we actually managed to hold some of the disasters together long enough — out of sight and out of mind — to fix them without anyone noticing critical systems were offline for months. IT was a bit more glitchy, slow, and unavailable back then, so the excuses were more plausible.
One article I recall in particular was in Scientific American some time in (IIRC) 1998 or early 1999. It prophesied (I use that word intentionally) that no matter how much money and effort was put into fixing the problem ahead of time, there would be all kinds of Bad Things happening on January 1. It called out in particular computers that were said to be unreachable, like hundreds of feet underwater on oil platforms. (Whether there actually were such computers, I don't know.) There was a sort of chart with the X-axis being effort spent on preventing the problem and the Y-axis being the scale of the resulting disaster. The graph leveled off while still in the "disaster" range, but still presented a clear message: "Give us more money and we can prevent catastrophes".
Somehow I haven't been able to find that article. Maybe SciAm suppressed it when the outcome turned out to be way short of a disaster.
There was also a TV (remember that?) news site or three that planned coverage beginning on midnight December 31 somewhere in Europe (Russia and China were off the map, I don't remember about Japan). Of course the news was that there was no news. (Yes, there were some computer programs that died or spit out junk, but nothing rising to the level of news.) I think it was an hour or two after midnight Eastern Time (US) that they ended the news cast.
Was there a Y2K problem? Of course. But it was largely taken care of before January 1, 2000, Y2K Jeremiahs notwithstanding.
Which only makes it more interesting. There are many takeaways one can have from this article, one of them is that:
- Problem X is serious.
- Y will address problem X
Is incomplete reasoning, or even an outright fallacy. Just because it's claimed that Y will address X doesn't mean it actually will.
Especially on high-stakes issues ("our business will collapse", security, safety) or emotive issues (social justice, security) this type of flawed reasoning seems to be a common problem.
I think it's either going to be a retirement plan for many who are young-ish IT people right now, or "optimists hoard gold, pessimists hoard cans and ammo" time with the pessimists being right. And a lot of this depends on how decision makers will remember Y2k.
(Don't reply with examples of things that use 32-bit time.)
A perfectly fine-tuned response would have a little bit more to fix on January 1. Of course, expecting society to perfectly fine-tune the response for something poorly understood is hard.
In fact, the REASON Y2K got so much budget and attention were the early companies that started discovering the issues and alerted the others. Notables include IBM, General Motors, Citibank and American Express.
Agreed it was a nice success. We also did pretty well in paperless office, the ozone layer and acid rain, automobile and airplane safety, and the war on cancer, and now obesity and diabetes.
You can't fix all bugs, so if the consequences really were going to be catastrophic then you'd expect at least a handful of catastrophes to sneak through, but that didn't happen at all.
No one is in a position to assert that. We have very little idea how fragile our civilization is. Perhaps it's pretty robust, and networks of interconnected problems (like Y2K) stand no chance of snowballing out of control. Or perhaps it's really, really fragile, and surprisingly little stands between us and a profound collapse.
It's very difficult to be certain, because it's such a complicated system, and one that we can't really test to destruction.
Would all the Y2K bugs have caused a widespread systematic failure if they'd gone un-fixed? Probably not... but maybe? Just like all low-probability, high impact risks, it's very hard for us to reason about.
How much money is it wise to spend on averting the risk of giant asteroid impacts? Hard to say. Probably more than you think, though.
I don't really like the whole popular understanding of the y2k thing. Substantial problems with infrastructure were fixed. It wouldn't have been Mad Max, but business operations could have been affected for weeks. Instead, we had almost smooth sailing because there was an appropriate amount of preparation beforehand.
(Not to say that there wasn't graft like he describes as part of that preparation).
Spent several weekends rigging up computers, and running a battery of tests on them.
Did actually find a handful of combos which misbehaved, mostly incorrectly handling the leap year IIRC. None of them had NTP running, so could have been an issue. A few didn't handle the transition well at all. Most got resolved by a BIOS update.
I recall thinking after 2000 came and went that sure, a fair share of superfluous work had probably been done, but the reason it seemed like such a nothingburger afterwards was because a lot of work went into ensuring exactly that.
I really hate this take some people have (especially people that weren't there at the time) that Y2K wasn't a real problem. It was a real, large problem effecting businesses across the globe. But the thing is, the problem wasn't that "everything was going to fail catastrophically at the turn of the year". It was that "some things could fail catastrophically at the turn of the year, and we didn't know which things would". So a lot of time and money went into figuring out which things actually had a problem, and fixing them. It was entirely possible that, if we didn't fix something, planes _could_ fall out of the air, or energy providers could shut down for long periods of time.
So we spent lots of time and money finding out what things needed to be fixed. And "it turns out that not that not that many things did really need to be fixed" is a GREAT result from that time and money; arguably the best possible result. But it doesn't invalidate that the time and money was spent, in the same way that "I didn't get into an accident" doesn't invalidate the fact that wearing your seat belt is a good idea.
I worked on a Y2K project in 98-99 for a major car manufacturer. They tested the ERP system for Y2K (in 1997) and it failed, in a way that would make the business fail. They replaced it via this project and the business continued to be successful without interruption.
Another company I know of simply backdated their whole system by 10 years. This worked, but unfortunately the contract cycle renegotation reminders then did not appear, and competitors took many of their contracts over the next year or so. They went out of business in a couple of years.
I implemented a new manufacturing and retail system in 1992, and we used 4 digit years. The company owner complained that they were a needless expense and annoyance and that there was plenty of time to deal with Y2K. They continued to use the system well into the 2000s.
The space shuttle never flew over New Year's eve/day as they couldn't handle a single year rollover, let alone a century. Some use cases can simply avoid the problem.
(IPL is "Initial Program Load", IBM-speak for a reboot.)
Same.
I worked on multiple enterprise projects for Y2K where we advanced the dates on the customer's system and ran it so we could see the magnitude of the problems and prioritize the remediation process.
There were some things that were temporarily annoying and there were some things that were critical to the business and could not be worked around manually by throwing people at it, the system needed to be fixed.
> The Andersen position was that “Y2K is a documentation problem, not a technology problem.”
Sure, there were other people at other companies solving the "real" Y2K problem, but the problem Andersen was solving was not so real.
I patched hundreds of vulnerable systems, rewrote code for mission-critical software and still ended up with a call-out to halted manufacturing plant on January 1st, 2000.
If there hadn’t been a global, coordinated response, things would have been a lot worse. And let’s not forget that there were life-changing effects of the bug beyond the snarky “garage door opener” issue mentioned in the article. [0]
[0] https://www.theguardian.com/uk/2001/sep/14/martinwainwright
But "almost all" is not "all", and won't be until there. It will certainly be interesting, as stuff will fail in a completely different way from Y2K.
I imagine a couple of days with the internet misbehaving everywhere is a safe bet.
But my real worry is embedded systems, where very little is 64-bit. Almost every embedded Linux system, IoT crapware, Android anything. Almost none of it will be able to keep time after 2038.
Are other Linux distributions doing the same thing? Sure, and there was a lot of work in the kernel (mostly in the 5.x series) to get it all nailed down. NetBSD and OpenBSD already tackled it.
But "all" can't be guaranteed, because people insist on keeping elevator controllers and industrial processes and anything which hasn't actually had capacitors explode and resistors melt running for an extra decade.
Sure, we say that now, but 584 billion years from now when we use up 64 bits, we’re going to be cursing our ancestors for being so short-sighted.
Expect lots of IoT or home security devices to malfunction. Or more critically industrial equipment, fire alarm systems, factory automation, etc. A bit over a decade a ago I helped develop an industrial spark extinguishing system. It's still sold and each setup will probably be used for 20-40 years. I know it will suffer from Y2k38, though probably (hopefully) only the event logging. When I raised concerns over this the answer from my boss was "I will be retired by then, don't spend time on that".
But...isn't capitalism supposed to fix this kind of reasoning? Turns out that what you really need is people caring for the right thing to do, and not just for money. Yes, often the right thing to do in a business context will lead to more money. But when time-frames expand and you are not the real owner...
As others suggest, I'm expecting a lot of the issues to be embedded code.
On a private forum I'm part of there was an engineer who ascended into upper management at a startup. He had an entire thesis that modern tech interviews were bullshit and could be replaced with short conversations and asking candidates to self-rate their abilities.
He posted long and confident writings about how he was going to save his company so much time and attract the best talent by having the shortest interviews and trusting candidates self-ratings of their abilities.
He disappeared for a long time until one day he came back and wrote a post-mortem about his hiring experiment. The quality of their new hires had collapsed. It was so bad that some of their previous top employees were quitting because they were tired of all of the unqualified people joining the company.
As he learned, if candidates sense that their interviewers aren't going to examine their claims too closely, many of them will inflate their claims as much as they think they can get away with.
In hiring, it only takes one candidate grossly exaggerating their experience and abilities to push the truly qualified candidates into 2nd place (aka not getting the job).
We all like to think that we can separate the liars from the truth tellers, but when you're trying to do it in extremely short interviews with questions going both ways, there isn't much time to deep dive.
... the Y2K Bug, and it prophesied that on January 1, 2000, computers the world over would be unable to process the thousandth-digit change from 19 to 20 as 1999 rolled into 2000 and would crash ...
That wouldn't be problematic, since the numbers don't loop around (like when going from 99 to 00).
I have been through another one of these where one part of a company changed their year entry from 20XX to just XX to save the people entering it time and multiple systems did not defend against this and suddenly started calculating all intervening years from the time of jesus up to the present day! Its was catastrophic, that simple changed caused several days of disaster recovery but months of defensive programming work as well. Date/time errors can be devastating to business.
I'm generally against rolling problems to future generations, but taking risks with a minuscule risk of going somewhat badly that may not even be within my children's lifetimes is not a core concern.
My sister's department reviewed every line of code in every product for date/time issues, and corrected them. But she was probably anomalous, by all the anecdotes. She ran a tight ship, and did things that actually needed to be done.
I wasn’t there for the play-doh management fad but I sure experienced capital A Agile and honestly can’t tell which is more infantile.
In a sense, the corporate environment of constant indefinite emergency described in the article is the same as arbitrary deadlines and non-stop “sprints”.
There were several issues that happened as a result of Y2K.
it prophesied that on January 1, 2000, computers the world over would be unable to process the thousandth-digit change from 19 to 20 as 1999 rolled into 2000 and would crashI also did my quick Lisp sanity test of saying (* 4 3) and expecting 12 -- but I got 14 instead. It was then that I learned the CADR speaks octal by default.
The biggest contingent of contractors came from a company of the AA universe that has survived the Enron scandal. My task was to advise on how to interface with a certain internal system. Easy job. The amount of actual code written was minimal and theses folks were big on generating class scaffolding from UML paintings.
Funnily, there was a constant rotation of contractors and they didn‘t realize I was internal. So, I was treated by them as one of their own. And it was six weeks of education that you can’t get in university or business school.
I learned the art of being billable which is miles away from doing any work. Matter of fact the hour didn’t have 60 minutes but a hundred. And whatever you faked doing, you did it with double digit precision. You documented that you documented someones proposed, ie. documented, code change that day. Two minutes spent, 1.89 hours documented. Yes, Github and stuff would only come the following decade. We are talking word documents here. Being billable meant that people would propose changes and retract the proposal with another one proposal. At least, the amount of side effects was limited.
I also learned the art of fake process. The contractors didn‘t necessarily know what they were doing. But they knew of doing things. So, lots of meetings were happening and random „specialists“ and „experts“ were flown around the planet to delight meetings with stuff they knew of. Having these people disseminate some talking points from recycled slide decks was .. uhm .. fashionable.
Third was the art of faking yourself. I vividly remember the guy coming in from Tuesday to Thursday who would bring and arrange the same set if golf balls at his desk. I must have asked him if he played or something. In the end it turned out some important dude made him caddy for an offsite golf event of some partners. And he picked up the balls that were lying around the course. Some consider that a big nono. But apparently it gave some status since every thought the guy had played at said offsite.
Highlights from the weeks being embedded with the contractors include a contractor-only dinner. No idea anymore how I managed to be there but some partner gave a rousing speech how they were at the fore front of innovation with this project. And that everyone frenetically clapped at the end. Memory has that the red wine was good, the food less so. Another highlight was an educative session by one of the flown in experts on what a FIFO queue was and how that aided in-order processing of arriving events. Frenetic clapping ensued.
Now this is all 25‘ish years ago and perhaps memories are more pointy than reality was. What is certain though is that the number of lines shipped into production was zero. Zilch. And that was not because someone realized all of it was fake. It was because the company went out of business eventually and was bought, sold, merged for its customer base and not its systems.
I've come to view the economics of tech this way too (although much more so true of contractors than employees). Workers are supposedly the downtrodden sucker class beneath BigWigs. But, if positioned the right way, workers can make out like absolute bandits, often at the expense of executives and majority shareholders. I've worked many useless, bloated projects that lined consultants' pockets before ending in layoff-inducing/executive-firing failure, projects that - ironically - were initiated at the behest of said executives themselves.
2038 Epochalypse countdown: 13 years, 4 months, 19 days, 7 minutes, 13 seconds
What? We discovered a perpetual motion machine at Y2K and I never heard of it?