Then after you go through a few of these, you'll realize it really isn't too bad and you've dealt with bad situations before and you'll gain the confidence to know you can deal with it, even when there's no one you can reach out to for help.
Then after you go through a few of these, you'll realize it really isn't too bad and you've dealt with bad situations before and you'll gain the confidence to know you can deal with it, even when there's no one you can reach out to for help.
- The software they purchased doesn't do anything because it was never staffed or configured correctly.
- Employees as a whole lose thousands of hours a year to terrible UX or meaningless requirements.
- Some capability doesn't actually do anything, but no one cares enough to notice.
- Useless meetings which waste collective employee hours daily.
- Capability serves no purpose, but just exists to meet audit requirements.
- Executives enrich themselves and waste company money constantly.
- Etc.
Downtime doesn't seem any worse than these problems, but it gets far more attention and panic. It feels a bit like terrorism vs. heart disease. It's important not to sweat it, because a company does not care about your sleep or mental health. It will push you as far as it can. I'm not calling companies malevolent, but they're a bit like a bully in that regard: they push you as much as you yield.
Fantastic analogy! One is scary but practically non-existent, and the other will bring early death to many people you know.
if no one noticed then can you really say it was broken?
My favorite anti-pattern ever that happens here is when you notice a bug that's existed for a long time, you fix it, then some other part of the system that had adapted and expected the behavior over the years breaks because of how they were handling the bug no longer works. Then of course the business comes to you like "why did you break this?"
Hmm… interesting thought. It’s interesting how labeling something a terror attack makes the public lose critical thinking and act incoherently for a bit. It’s like a stunning shock response. The very same this author describes.
Do you think some managers intentionally play up the severity of problems to push their teams into this panic mode?
People in customer service know clients definitely do.
It is interesting how these concepts connect.
I'm sure some do. Much more often I observe that the managers themselves are just reacting to the incentive system in their environment -- and that those incentive systems are complex, not-widely-understood, and often built "accidentally." ie, companies are often accidentally incentivizing the wrong behaviors and are often powerless to prevent this.
It's difficult to understand sometimes, but they are pressured from above, and they don't have the means to fix it by themselves.
On a more positive note, other managers will have a rapport with their teams where if the team only learns of a problem, it will be addressed. They only need to act as liaisons — stick something on a kanban board somewhere, and it will get resolved. No loud crying necessary, just business as usual.
For a website - particularly ecommerce websites - you have many choices. Rarely is a product only sold on a particular website. Being down when someone is trying to place an order can and does result in losing the sale, and potentially the customer forever. Customer loyalty is often fickle - why would they wait around for you to fix your stuff when they can simply place their order on Amazon or another retail website?
For example the classic thing is to always have minimum of 3 or 5 nodes for every stateful system. But in some companies, 1 hour of planned downtime on Monday mornings at 7AM to 8AM for operations and upgrades (which you only use when you need) + eating the times when the machine actually dies, would be less downtime than all the times you'd go down because of problems related to the very thing that should make you more robust. An incident here because replication lag was too high, an incident there because the quorum keeping system ran out of space etc and you're probably already behind. And then we have kubernetes. At some point it does make sense, when you have enough people and complexity to deal with this properly, but usually we do it too early.
From an ecommerce website's perspective - it is exactly the situation most of us are in.
Unless you are the manufacturer for your own goods, then you are competing on often-thin margins within a crowded industry with lots of competition where everyone more-or-less offers the same price.
In an age where more and more people don't even consider shopping anywhere but Amazon, being down exactly when that customer decided to grace your website is akin to giving yourself a nice, stinging papercut. Do this often enough and long enough, those papercuts will start to leave scars.
So ya, being down is unacceptable for most ecommerce websites. The website literally generates your revenue, and without it you receive no revenue.
That's not true. You don't have to manufacture your own goods to not compete with others at a commodity level. It's quite literally why branding is so important.
But I do find it funny you wound up doing exactly what the above commenter stated, and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while.
Most of the goods you purchase are from brands not owned by the retail website. Pick a hobby and consider the prominent brands, then consider the prominent websites people in that hobby often patronize - they are very often not the same.
> and are acting like this is a world ending event for a small-medium sized ecommerce site to be down for a while
Because it kind of is. The smaller you are, the more significant of an impact an outage has. Nobody is going to think "this website I've never heard of totally isn't working right now, so I'll just wait around until they fix it". No, instead, they will go on to their next google search result - or Amazon. This hurts even worse if they clicked one of your ads while your website is down... costing you actual money and not just the sale opportunity. A customer that might have had an expected lifetime of 1-3+ orders now becomes 0.
Websites don't have "business hours", and routine maintenance is not acceptable for customers that shop when it's convenient for themselves. Remember, ecommerce exists as a better way to do over-the-phone and catalog/mail ordering. One of those benefits is you can order whenever you want - so it is in fact important for the website to be online at close to 100% as possible.
If it was groceries, I'd go to the store.
Unless in-person-purchasing is an option, I'm just going to wait for my preferred vendor. My account's already set up and clearly they have what I want.
By the numbers, most customers are shopping on any particular given ecommerce website for the first time. It's expensive and difficult to gain new customers - and even more difficult to gain repeat customers.
What would you have done if you had not purchased from ebikes.ca before and just happened to stumble across them after a google search? What would you have done if after clicking the ad/link, the page didn't load? Would you try again tomorrow? Probably not... that is the issue. Instead you would be naming some other small business that was online when you needed them to be.
> What would you have done if after clicking the ad/link, the page didn't load?
archive.org first just to check if it's down temporarily. But if you're a vendor just reselling Alibaba, then there's not a lot of reason to shop from you. But with real value add, you can get customers. That's exclusive products, support/warranty, niche product, etc beyond just "fulfillment".
There's an entire ocean between Alibaba flippers and strongly branded products sold first-party by mega-corps.
Consider the physical space and walk into a Target or Walmart - how many products on the shelf are Alibaba flips, and how many are sold by the manufacturer? Target has house brands, but Dawn Dish Soap is not one of them. You can get Dawn Dish Soap from most stores - if one day Target was randomly closed are you going to sit out front and wait for them to open, or are you going to another store?
Back in the digital space - let's pick golf... how many websites do you think sell Callaway golf clubs? How many of those websites are the actual manufacturer/brand? One? A google search for "buy golf clubs online" yields probably close to 50 websites just on the first page alone... of which 80% appear to be small businesses.
You start clicking through the search results looking for deals - and some of the pages don't load.
You're telling me you're going to go to archive.org and lookup the history of these random websites, then wait for them to come back online just to see if you want to buy something?
I don't buy this argument at all...
> That's exclusive products, support/warranty, niche product, etc beyond just "fulfillment"
People really don't consider or understand supply chain logistics, distribution and it's relation with retail... people really like to casually wave their hands and assert resellers/distributors have no space in the world, yet they buy products from resellers/distributors every single day without a moment's thought (Target & Walmart!). The "value add" is making desirable products accessible to regular people.
Regular people aren't going to pony up a $75,000 Purchase Order, fill a shipping container, and wait 6-8 months transit just so they can have a new cell phone case or new bookshelf. That's where distribution networks come in handy. We buy the $75,000 container so that you can buy just one item.
I guess we can't really blame people for not understanding this - people rarely understand how their beef and carrots got to the grocery store either.
> sit out front and wait for them to open, or are you going to another store?
Many times I've wanted to get something from my preferred grocery store, but they're closed so I... just went a different time.
Most people aren't going to shop a bunch of random websites to buy "Callaway golf clubs". They're going to buy it from Dick's or Amazon. And if those websites are down the moment they check, they'll just wait an hour because it isn't urgent or figure it's a problem on their end.
Fuck, b&h photo takes their website down a whole day every week!
> I guess we can't really blame people for not understanding this - people rarely understand how their beef and carrots got to the grocery store either.
Thanks for the lesson, I learned how smart you like sounding.
>> sit out front and wait for them to open, or are you going to another store?
You were asked if the physical store was closed - because that's the analog you are deliberately overlooking.
> Thanks for the lesson, I learned how smart you like sounding.
Some people unwilling to learn. Even worse, some people insist they understand an entire industry, having spent exactly zero time in that industry. The hubris on display is amazing.
depends what you're selling, and there's always the chance that someone is up at 4am their time, but if the page was down from 11pm to 6am every day for a mainstream physical product, they'd probably not lose too many sales.
unfortunately, losing a sale puts a business into a tizzy, to they tend to react poorly to that thought. it's because it's easy enough to leave the computer running unattended over night that business hours aren't a thing that websites are in that spot, but watching the traffic flows while on the Internet traffic team at Google on top of some emotional development, means that for a small bespoke business, SLAs don't even have have any nines for that business to remain in business.
we're biased to by the very nature of us commenting on an Internet website, but there are still customers out there that don't use the Internet on a daily basis.
I think this is what OOP meant about "zero-downtime culture" being out of control. Unless you're a hospital keeping people on life support, I don't want to hear about zero downtime.
You build a culture of nobody caring about the singular thing that brings revenue into your organization. You're fine with shareholders taking a hit, because it's not your bank account - but what happens when a company has to let people go because they don't have enough customers and orders to support everyone? Oh, now they're supervillain out-of-touch CEO's I presume?
> Unless you're a hospital keeping people on life support, I don't want to hear about zero downtime.
Then look for other work? If you are being paid to keep a website online 24/7, and it goes down "for a few hours" and your attitude is "nobody died", then you're simply not cut out for that job - find another or be terminated.
There's not a single person on pager duty (or whatever) that doesn't know what they are getting paid for. Complaining about a job you volunteered (at-will employment!) for is insane.
You seem to be stating that any loss at all no matter how small is a complete disaster without really qualifying it. Yet one could easily make the point - well, what about the customers you don't serve at all because they want to buy a rucksack and you don't sell them? That number could be far higher than the ones affected by outages.
I think this probably just depends on how big the numbers are to be honest. Yes, if 1% of people are millions of dollars, then sure, we care. Usually it's not anything close to being like that.
Not sure if it's quiet-quitting or what, but the last few years (basically since covid) my consumer experiences in the US are feeling a lot more.. shall we say, European? And I guess that's fine, pretty much everything really can wait. But it's hard to maintain one's own sense of professional urgency to keep things smooth for others when literally nothing is smooth for you.
I would be furious. If I spend $10k I want to be speaking with a human if something goes wrong. I feel like in the US we have given up on decent customer experiences.
"Phoning it in" is when your heart isn't in it, you're just reading your lines or going through the motions.
Without information, it is easy to assume incompetence, or worse, just not caring.
Funny story though, veering a little OT, but recently I had to haul ass from one end of the airport to the other due to arriving on a delayed flight with a connecting flight with a different airline. As I was about to enter the airplane, after getting my boarding pass scanned, completely drenched in sweat, the flight attendant saw me and said the captain wanted to see me and ushered me into the cockpit. I found myself in the cockpit, where the captain proceeded to talk to me in the local foreign language while I was thinking, "Wtf, isn't it illegal for me to be in here after 9-11?" I then started repeating, in the local language, that I didn't understand them. But that didn't have the desired effect intially and they kept talking. I repeated myself both in their language and English until the captain finally said, in English, "Are you a passenger?" When I said yes, he told me I could sit down.
When I sat down, the only rational explanation that I could come up with was that despite being in civilian clothes I was mistaken for a pilot. Not just any pilot, but one of the pilots that was supposed to be in the cockpit. I could only conjecture that he was a no-show and that they had to bring in a substitute to replace him which caused the plane to be delayed and allowed me to make the flight.
Any commercials pilots are free to correct my -- which normally should be a very silly -- theory. If it could possibly be true though, being the no-show pilot in that situation might be a little stressful.
Downtime for these is critically impacting people, and you're damn right you should explain what went wrong and how you'll avoid this in the future. Because the only reasonable alternative is nationalizing them. "I dun fucked up, but meh" is not an acceptable attitude.
I'm cool with it. I don't care that some hard drive went bad and the monitoring system didn't pick up on it because some queue was full. Who cares? If this isn't happening constantly, I assume they're going to learn their lesson.
That's pretty much the opposite of "but meh". Absolutely, mistakes happen, they aren't the end of the world, but they need to trigger a learning process. Looking at airlines and ISPs, that process is, in the most charitable viewing, processing glacially.
One time a few years ago, a coworker called me to tell me the stress we were under was causing her hair to fall out. I mentioned I'd lost significant weight and wasn't sleeping well, especially since another coworker had quit and now I was shouldering his responsibilities (programming and managing a junior). She said she was going to quit. I wished her the best. Around six months later, I had to quit too. It took around a year for me to get back to normal sleep patterns and for my heart to stop racing in the night.
Oh yeah, another guy from that team had a "cardiac event" due to stress and had to take a month off.
THIS.
Worked for a huge corporation when I first started contracting as a developer. This was a daily occurrence. Someone would push some stuff to prod and it would break their very large e-commerce site. Sometimes it was minor stuff, other times it would cripple the entire site. Most of the time, nobody GAF at all.
I remember sitting at my desk one day and my senior dev walking by my desk like Lumbergh in Office Space. Coffee in hand and just casually said to me, "Looks like someone broke the build, most of the video game pages are throwing 404's right now, going to lunch, see you soon." and then just walked away.
I also found out one day one of the CDN folks that was running the Akamai account quit and then a month later, we found out one of our secondary content servers had been down since the day he left. When dude left, he never transferred any of his knowledge to the other team members so once they found out the server was down, it took another two days to get in contact with someone at Akamai who was handling our account.
At my previous job if something went down on our site, it was a four alarm fire, war room, and all hands on deck to get it resolved or heads would roll. It was so dysfunctional to work somewhere when something broke or stopped working, nobody was in any hurry to fix it. Several times I just thought, "Is this what they mean when they say "the inmates are running the asylum""?
That's funny, because reading your first sentence I was thinking that was the dysfunctional place. I had a boss that would take problems from 0 to 10 in a flash, when the problem was really a 4 or 5, and it really was not a great environment.
One of my programming mantras is "no black magic." If I don't understand why something works, then it's not done.
I take this same approach to an incident. If someone can't coherently identify why their suggestion will have an impact, I don't think they should do it. Now there may come a time that you need to just pull the trigger on something, but as I think back I'm not sure that was ever the case in the end.
It was wild to see the top brass—normally very cool and composed—start suggesting arbitrary potential fixes during an incident.
I'm willing to throw shit at the wall early in the triaging process, but only when they are low-impact and "simple" things. stuff like -
have we tried clearing cache?
have we checked DNS resolver for errors?
have we restarted the server?
etc. I try to find the "dumb" problems before jumping to some wild fix. In one of the worst outages of my career, a team I was working for tried to do a full database restore, which had never been done in production, based on a guess. At 3am on a saturday. I push back really hard at stuff like that.
so many ppl nowadays don't care about knowing how thing works, they just push shit and cross the fingers.
This. * 1000
I am very grateful that my earliest training was as an RF bench technician. It taught me how to find and fix really weird problems, without freaking out.
However, it does appear that this particular event may have been caused by reliance on a dependency that may not have merited that reliance.
I'm really, really careful about dependencies. That's an attitude that wins me few friends, in this crowd, but it's been a long time, since I've had 2AM freakouts.
But debugging RF stuff is hairy AF, so it’s a great forge for making problem solvers.
Imagine how much more the damage would have been if someone was telling this solo developer to stop trying to make 'fetch' happen.
You're absolutely right that fear interferes with one's ability to reason rationally but I'd also add that fear is highly infectious. If ICs sees their leaders/managers/colleagues panicking, they will often panic too. Thankfully my VP was always level-headed and prioritized clarity over action.
Apparently you worked for a Navy Seal, so yeah, likely a very cool dude :)
We're not doing open-heart surgery or launching people into space. They can wait 1-2 hours extra.
Earned a lot of my respect on that day :)
You are more likely to regret taking large debts or missing out on life milestones than on losing the job or going hungry.
Just saying -- kids pick up languages real fast, friendship treaties and tax benefits are abundant and with 2M you can shop around for a better lifestyle if you really want to not have that stress and fear, unless you are determined to become filthy rich, I ofc don't know you.
I'm constantly surprised how many well-off people try to pass themselves off as "one of the poors". My family and friends do okay with a net worth way below that, in the Bay Area. I've worked minimum wage jobs here in the bay, and had co-workers actually struggling to make it to the next paycheck. It really gets on my nerves when someone in software chimes in with "we're all struggling"—No, we are not all struggling on the same level. Guess how much the person making your fast food or working at your local grocery store is making... And think about how they still live in the same city as you.
"$2M isn't that much". Man, I can't imagine what it takes to come to that conclusion.
I just drive a regular, normal Lamborghini - like my father, and his father before him.
As someone who was layed off and went without a job for 13 months (tech bubble), I am _always_ scared about my financial situation. Sure, if I lose my job, I have the resources to hold out while I find another, but
- It _will_ impact my lifestyle, because I'll need to be more careful about what money I spend
- It _will_ make my family unhappy, for the same reasons
- I've been in the position where it's "do we eat or pay rent next month?", and it sucks. I likely won't get back there, but the fear of getting there persists.
Look, I get it, $2M is a lot of money. But there's a fair number of people that can get to that amount (especially living in an area where a 2br dwelling is easily $1M of that), still be saving _more_ money, and still fear the idea of being out of work for months. Especially if it's a single earner household. And especially if there's children.
Yes, it will impact your lifestyle, and you may have to be more careful about what money you spend, depending on how you're currently spending it. So what? 99% of people on earth live like that, and most do just fine.
The fact that having to live a non >$2M net worth lifestyle would make your family unhappy sounds concerning.
2. The vast majority of people, if they suddenly had to live with $0 income for a while, would be concerned, regardless of how much they make now.
3. The vast majority of people, if they suddenly had to live with 1/2 their normal income, would be concerned, regardless of how much they make. People make life decisions based on their income, and having that thrown up in the air generally causes people worry and stress.
The phrase "$2M net worth lifestyle" does not mean anything.
A 2M net worth could be a 1.5M house (which in the bay area is a simple nondescript house) and 500K in a 401k (which you can't touch until you're 60, so it's off limits). That's it. Neither of these can be used to pay any monthly bills.
So this person with a 2M net worth (locked in assetts) is still 100% dependent on that monthly paycheck to cover all expenses.
There are literal minimum wage workers getting by in the bay. I really can't understand this obsession with not wanting to admit that you are well-off.
And there are people that don't have a job and live off the land. Each type of person has different expectations and commitments.
- The person making $80k/year is well off compared to the person making nothing because they can't hold down a job; but they're still reasonable to be worried about losing their job, being unable to pay for their apartment, and being out on the street.
- The person making $150k/year is well off compared to the $80k person, but can have the exact same fears. Maybe they're worried about losing their house because they won't be able to pay their mortgage.
It can be very hard to pay your bills after being out of work for a year, even taking minimum jobs to help make ends meet; because people make commitments based on their earnings. And even _reasonable_ commitments can be rough if you're out of work long enough.
> I really can't understand this obsession with not wanting to admit that you are well-off.
The poster with $2M in net worth never denied they were well off. You're the one that seems to be denying that it's possible to be well off and _also_ be worried about the impact of losing your job. Your stance is... mind boggling to me.
In any case, I would suggest saving up an emergency fund to the point that you are not scared of losing your job. I understand making that people must make commitments (car payments, mortgages, student loans, etc.) and that the risk of not paying them off is always there. But I would hope that a millionaire of all people has enough collateral for a loan if necessary, and that they would have good enough credit to be approved.
If you have your finances in order, you should not be scared of losing your job. Having actual fear from the thought means that maybe your assets aren't working for you (they are all deprecating assets or the cost of their upkeep is higher than the return). The one thing that can and should terrify you is illness or injury that makes you unable to work.
As someone who was out of work for ~13 months during the dotcom bust; this is absolutely false. Sometimes, things work out such that getting a new job is more complicated than just having a good skill set. And honestly, the idea that "if you can't find a another job quickly, you're clearly not very good at your job" is flat out insulting.
> I would suggest saving up an emergency fund to the point that you are not scared of losing your job.
13 months eats a lot of runway.
To be clear, I was only referring to leadership or management roles (given the context of a millionaire in this industry). At that point in your career I would expect one to have many career connections and/or enough sheer charisma and experience that makes landing a job doable if not a breeze.
It's not meant to be particularly insulting, just realistic.
> As someone who was out of work for ~13 months during the dotcom bust; this is absolutely false.
I didn't consider exceptional events such as the dotcom bust, or the 2008 financial crisis or the Coronavirus pandemic and subsequent inflation. If we include those, I do agree that if makes sense to fear losing your job.
I do concede that we've had multiple "once-in-a-lifetime" industry upsets in a very short while and with that context it could make sense to panic. But in normal circumstances, I still think fearing job loss as a millionaire comes from somewhere more emotional/instinctual than rational.
Man, some people here are completely out of touch with the average Eathlican. I'll have to show this discussion to my friends.
With that kind of money you could just... you know, move pretty much anywhere on the planet and live comfortably for many years, until a good opportunity comes up. Depending on the place, it might last you the entire lifetime (see my salary above).
The amount of people I've seen on here saying "It's just not an option to quit Google/Meta no matter how awful they are, I have a mortgage to pay!" even when the tech job market was still red-hot is quite incredible.
It’s absolutely enough to live a frugal retirement on, but not some sort of massive nest egg. You can safely withdrawal about $80k/year from that without it diminishing.
Don’t get me wrong, that’s the average income in the USA, but most deep career professionals would be taking a step down in terms of purchasing power. No real source of income means you can’t really finance anything. Major purchases are essentially pulled straight from that nest egg.
My net worth is less than yours. I could live on your interest fine and just retire.
Not really for most scenarios.
For someone who is single and has no kids and plans to stay that way forever and is happy to live a very simple lifestyle, perhaps. Anyone else, no.
You'd still need to buy a house even if you move to a real cheap area, so you don't have 2M in cash. Then, how are you paying for health care? Kids education?
Health insurance can be purchased from healthcare.gov in largely the same way home insurance works.
$1.5M would get you $60k/year without ever touching your principal.
That being said, waiting even a few years drastically changes the math into your favor. 5 years of a job, plus interest on your retirement fund could easily double it.
Likely-time-remaining is a huge factor in figuring out how much it’ll take to retire at a given level of risk (of running out of money too early).
Alternatively, if you have $2m in stocks that are growing, you could consider the growth amount as potential income, because you could always sell it. Generally the growth is better than 3%, but down times will also hurt worse since you might have to sell at a low.
Or you could purchase a residence with $1m, and buy dividends with the rest. Taxes on the residence will be about $10k, with dividend income of $24k, leaving $14k for food, etc.
You won't independently wealthy in any of these scenarios (not in the Bay Area, anyway), but in these scenarios you don't actually need much from a job, maybe $40k with a family. So you should be able to save several years worth of that on a Bay Area salary, which leaves you with quite a long runway.
And if things get really bad, you can move to the middle of nowhere, buy a house for $300k and actually be independently wealthy.
3% is on par for SCHD, which is a popular ETF dividend fund (that provides income, but also growth). Different risk profile than treasuries though.
Don't plan a retirement around a HYSA at 5% because those rates will go back down.
If the company wants to derisk their business, they can hire more people and create systematic safeguards in their services over a long period of time. The return on this investment is theirs. If they don't make this investment, they don't get the returns - simple as that.
If you are afraid of "losing your job" - remember that you might lose your job even if you were the hero because of reasons not really in your control.
The only time to really panic is when your family is under crisis. To prevent your family from going in crisis, you should build safeguards in terms of health, income etc.
Business owners can do the same if they want safeguards. If not, fine, things will just break. And it is NOT an individual engineers problem to fix all of their poor decisions.
It seems to relate to fight or flight, as opposed to freezing. All three are not great and can significantly escalate a bad situation.
I’m talking about the brain fog more than the action itself, though. I no longer have it since I’ve been trained in emergency field medicine. That training course gave me enough exposure to stress to no longer get into this state of confusion. But I want to help others deal with it and it’s hard to find info online.
Another panic reaction is selling off everything at the first sign of trouble. And some people hold on to doomed stocks for way longer than what makes sense (look up "Intel guy" on /r/wallstreetbets)
Day trading requires due diligence and nerves like steel, else you are just making somebody else a little bit richer.
Maybe more specific though, situations where mistakes cause more dire mistakes are sometimes called "incident pits". Not unknown to engineers, but I think we ripped it off the scuba-divers, mountaineers, and other folks that are interested in studying safety-critical situations. https://www.outdoorswimmingsociety.com/the-incident-pit/
> We must do something. This is something. Therefore, we must do this.
Yes, literally cooling down. Combined with conscious breathing. And then go on doing, what you can do.
We all slept so much better after that
My first serious job as an adult was avionics repair. I knew how to do SOME component-level troubleshooting but I wasn't very good at it and always got lost in schematics pretty quickly. When I ran across a broken device that had a problem I couldn't solve within 30 minutes of investigation, my first instinct was to say, "welp, this is beyond my abilities" and ship the card or box off as permanently broken, or try to pawn it off to a more senior tech in the shop.
Fortunately, I had a mentor who disabused me of that tendency. He taught me that EVERY fault has a cause. Every time you think you've exhausted all options, what it really means is that you've only exhausted all CONVENIENT options and you just have to suck it up and get cracking on the harder paths.
Afterward I made a promise to myself to never get into such a panic again. But funny enough, when I feel like panic is immanent I just remind myself that whatever problem I have is absolute peanuts compared to that event. So for me at least having one truly bad event helps me put things into context.
(1) the logs had JSON-escaped JSON and I am slightly proud of those regexes because it is the only time I hopefully ever have to use 8 backslashes in a row https://xkcd.com/1638/