Suits ignored IT's warnings, so the tech team went for the neck
theregister.com
theregister.com
I made sure all the tickets opened by our clients went to the people who wanted to keep the old solution. It worked, we moved to our system and we rarely have issues with it.
Instead of the organization just doing the work to stop the pain, they throw bodies like so much human analgesic at the problem - I hope you'll let me torture this metaphor some more - and become addicted to this state of affairs.
Go above and beyond, sure. But never ever be an invisible martyr. If it’s management’s mistake, management needs to feel some heat or they won’t learn.
Which is basically what you've written. This concept should be discussed more often.
Really shows the limits of an individual when their livelihood depends on the source of the problem
If they don’t change anything, don’t suddenly pull all-nighters to then meet the deadline. A bad timeline was never your mistake to fix. And assuming you’re a productive member of the team, you’re not going to be fired for helping management plan like this.
At a macro level this is exactly why we're seeing such a strong disconnect between political/corporate leaders and "regular" people. The ones up top who make the decisions don't feel the pain of bad policy because they're so sheltered from day-to-day reality of most people
It happened a lot affecting our personal lives significantly, waking up in the middle of night just to reboot/restart some service.
Over time, the whole team started ignoring on-call alerts, and worked real slow. Pretending that our internet was down or laptop is not booting up. Longer it took to resolve alerts, our automated systems would start sending alerts to higher up in the chain. Also it started to impact our SLA and other metrics.
Finally, they decided to allocate resources to fix and stabilize the system.
And then it reinforces the management's conviction that they are absolute geniuses.
The teams sat next to each other but didn't speak.
The legacy team called in a bomb threat to a package left under the IT director's desk. It was taken very seriously. While their cause was already lost, that didn't help them personally.
The new system team eventually called Oracle, who came and re-wrote their queries.
The real issue is that middle management has a ton of politics that muddle up everything. I can say to my team that I messed something up and that we have to fix it; but I've seen people higher up who will bend over backwards just to not have to roll back a decision (and sometimes not even a very important decision).
So maybe someone had already sold to the boss that the equipment was good for another 10 years.
The answer to “why are we planning to spend so much time on this non customer-facing project” isn’t to just dive in and start talking about Kubernetes node pools and pod affinity rules, it’s to talk about how we’ll be able to cut cloud spend by 20% right off the bat while improving site speed, which is important because churn is up 25% YoY and a majority of recently churned clients have complained to support about slowness before cancelling.
So frequently I’ve seen former, then the engineers go off in a huff that management won’t support their project, and management goes off wondering why engineering is screwing around while customers are churning.
companies get so fixated on having their engineers do "engineering work" that they don't communicate what's actually going on at the business level.
Shouldn't management communicate? Shouldn't management manage?
In all my career, I can think of maybe a handful of non-techs whose communication was measurably better than grunts, that could manage better than chimps throwing darts.
I will concede that most tech's, such as myself, fail miserably on other social niceties crucial for royal politics (and the money chase). Such as deception, stealing valour, skulduggery, sucking up, ambiquitiy, enthusiasm, and fashion.
At some point, you've got to think about whose role it is to manage who.
If your management needs to be spoon-fed the decisions they must sign off, and reacts poorly to signal that is not perfectly worded or inadequate because the relevant information was not transmited down, what use are they?
Information needs to flow both ways, and in so many situation, the reason information pushed up is not relevant is that context wasn't pushed down.
Them lacking context or not is a separate problem. But you should be able to make a case for your proposal without that much context. What are values, risks, known, unknowns, costs, etc.
The original claim was that engineers tend to have poor upwards communication skills, fueled by low understanding of management goals. My claim is that that situation is to be expected when so much information doesn't flow down.
> But you should be able to make a case for your proposal without that much context. What are values, risks, known, unknowns, costs, etc.
...And that case will often be deemed irrelevant to the company's goals, or incompatible with the company's challenges (lack of funding, staffing, etc).
Then you also pointed out that many managers can't spare the attention for some decisions they are asked to do. Maybe this kind of decision shouldn't have been pushed up in the first place? Stories of engineers ranting about having to climb 2 ranks to approve a $100 expense are pretty common, for instance.
Was that a made up example? If not, is there a successful case study somewhere you can cite?
After we moved to k8s, the release model went to "every dev can use kubectl to deploy" and it took a minute or ten to release something to prod. Needless to say, we all loved it much more than the previous setup. Even more so after someone whipped up a script to automate all the kubectl invocations.
One of my former colleagues wrote an article about it here: https://wetransfer.com/engineering/migrating-frontend-to-kub...
There's existing tooling for that : Skaffold, ArgoCD ...
When we were first releasing this Helm charts were considered pretty unproven and AWS didn't have a managed k8s offering yet. It was very early days.
However that's not the problem described in TFA. It was clear what the user-facing implications of not upgrading the line were, and management didn't care. Sometimes people aren't rational.
Regardless, who knows whether this was the case in the story being discussed, but it’s not unthinkable that this genre of communication breakdown was involved to some extent.
Example:
Why do we want to spend money on Jetbrains licenses for developers? Because we want them to be happy, comfortable, and productive. And why do we want them to be happy, comfortable, and productive? Because they build stuff that customers want.
o Talking to HR: "We need to do X to reduce/increase headcount needs".
o Talking to legal: "We need to do X to comply with Y regulation or decrease our exposure"
o Talking to other teams: "We need to do X because our current system can't scale to take your load".
o Talking to other teams that don't want to work with you: "We need to do X because your system can't scale and we're going to drop our dependency on you".
o Talking to the business: "We need to do X to save $$$ this year"
But if you simply say "We need to do X because of [highly technical reason]" then no one outside of your team is going to care or help you.
They shouldn't have asked management for money, they should have presented management with two alternatives: pay $x now to upgrade network infrastructure, or pay $y later in rush fees, IT overtime, and customer loss due to degraded UX. Then you let them make the decision.
It's absolutely possible the business was in a temporary cash crunch or needed the money to get a more important revenue-generating initiative past the finish line, and eating the higher costs later might make more sense to the business.
It comes from a difference in motivation, in what the different "tribes" believe to be important. Bridging this gap is more about translation and seeing the others' perspective than anything else. This is the reason project managers exist, and why their existence is mostly miserable.
We need to acknowledge that every layer of this cake thinks they're the most important, but in a de-facto dictatorship fuelled by customer spending, the lines of power are closely aligned with the flow of money.
Tech decisions are approximately irrelevant to the people closer to the money stream, and they're primarily evaluated in terms of potential profits and actual costs. In other words, tech only becomes important when it facilitates more profits or incurs unexpected expenses.
However, and that’s the big gripe, each and every one IT proposal I see costs money. It not just costs money, everything costs money, but it raises the long term average cost per unit. Even when we are dismantling ancient and totally non functional systems, the TCO of the modern solution is higher than the previous. Look I’m here on HN, can program a bit, understand quite large parts of the tech stack. But not why even highly capable IT-staff and management whom I have confidence in can’t seem to explain their contribution to the basic underpinning of business: average cost should go down over time and with scale.
(Advice about how to talk about this with IT people is more than welcome! A lot of organizational distrust and poison sadly follows from this.)
One of the more current examples is that IT-staff forgets that they are 2-3x more expensive per FTE than general workers. So when I see a plan and calculate how much middle management should save in reduced workload thanks to plan X both sides don’t like the outcome. The end of the day picture that companies exist by grace of customers willing to pay for your product (and cost loading!) is lost at that moment.
But sometimes, management is: https://www.youtube.com/watch?v=r8miwsWtzRw
When was the last time a manager asked you what you needed to do your job better?
If someone told the boss the equipment is good for 10 years, maybe you can work with them to come up with new requirements the old equipment can’t meet, and let them present those requirements to the boss, so they have a reason to support instead of opposing you.
The alternative to this sort of political shenanigans is to be ineffective, or to work in a small enough company that you don’t need to.
What happens more often than not is that the C-suites and their underling managers just don't give a shit. They've got their "grand plan" that works in their minds, and no amount of annoying devs that tell them that their fantasies can in fact, not, be made into reality can sway them otherwise. Doesn't matter how well you explain it, they want it done, so it'll get done, regardless of any other problems.
Hell, they perfectly understand what we're talking about most of the time, it's not like we're hitting them with obscure jargon known only to the highest caste of the nerds, it's really just that they don't care. I do get where they're coming from as well, as we can be extremely annoying oftentimes (at least, I know I can be extremely annoying), but the feeling is often mutual when I'm given a ticket I've argued vehemently against, more often than not with good reasoning.
I just wish more companies weren't being run by brainless, heartless MBA types and there was more of us engineers in charge, but c'est la vie.
also from the perspective of the aforementioned suits, they can hire IT guys easily, who cares if one of them gets very uppity and quits if we blame them for everything/anything.
Corporate IT grew from office clerks. Obviously you don't need strategic planing to see that fax machines and PC's work, so corporate IT started as being nobody important.
“The current growth trajectory means we’ll be out of capacity in a month, and it takes 3 weeks to install more capacity, so you’ve got a week to find the budget. If we don’t do this, customers will experience a slower service which may lead to complaints and reputational damage.”
It’s also possible that the execs understood perfectly and were happy to accept the risk of the service being slower for a short period of time. Maybe the Internet-using customer base was tiny, or the service in question wasn’t critical, or the execs were handling general financial pressures leading them to want to sweat every asset until it was impossible to ignore, or maybe they just understood that “the internet being slow” was normal for the 90s and it would be unlikely to hit their reputation in the long term.
The suits are cheap or dumb or it’s their fault they can’t speak tech. But often there’s just a lack of communication skills.
Business execs want everything broken down in "how much will this make me" and "how much will this cost me". The problem IT faces is that sort of impact isn't easy (or is impossible) to articulate. We can easily speak to how much the fix will cost, but not to how much it will save or bring in.
This is why orgs are terrible at security. How much will fixing the log4j issue save? Who knows, maybe nothing, maybe the entire business.
The budgetary control management system that companies like this use isn't primarily designed to limit spending to the minimum necessary. Its primary purpose is to prevent any political power being built away from the management office, by constraining access to the critical resource.
I mean this literally and historically. Budgets being used for management control dates from when the CEO of General Motors couldn't control the sprawling divisions they'd bought, so he (well, his CFO) brought in budgetary controls invented by McKinsey (the man, not the company) so they couldn't move without his say-so. And of course because it worked for GM, it must be good for everyone.
Maybe it is hard to communicate with people who lack a fundamental understanding of the critical tools they use.
If you don't understand how to write software it's like not understanding how to run a newsroom for a newspaper - you simply won't make the right decision often enough no matter how clever you are
It may seem like a ridiculous idea for a floral delivery service. But if you know your people enough to trust them or not trust them, you don't have to be an expert in everything to make the right decisions.
Which brings us a full circle to the nerds hating the suits because they want to quantify the unquantifiable.
OR, you could just go with what the experts say and be tagged a pushover.
If we think of management as "coach / therapist" (ie the eight rules of google management" then yeah, the goal of management is to take a team of high performers and help them not to implode. But I don't see that as "management"
I think there are better definitions
And posts like this where you act so smug and callous do nothing to change this.
That doesn't sound like a failure to communicate.
One day, a major disk crash occurred, and the whole company went frozen for a couple of weeks (couldn't fulfil orders; couldn't manage support requests; couldn't make any proper commercial proposal; couldn't find a customer's number or address from the database; etc). IIRC some disks were even sent to Kroll Ontrack. After that disaster, of course, a proper backup unit was bought...
Months later, a student developer runs a delete query in production with the where clause commented out. Oops, an entire table got wiped. Well at least there are backups... except a background job noticed and quickly created 4 years of invoices and emailed every past customer to pay again.
A test environment arrived soon after.
No backups, the tape drive was busted. No replacement parts for things like the bleeding edge 10mbps NIC.
I kept pressing that we need to replace it or migrate off of it to a cloud server, kept getting denied. "It's not in the budget" "It would cost too much".
I mean, we're talking a few hundred dollars a month for the minuscule usage we had, compared to the massive risk that if the system went down it would cause the entire college to collapse and possibly be permanently shuttered.
Finally the ball dropped and it went down. For days you could access it at the main terminal but it would not speak to the network at all.
People are starting to panic. Talks of bringing in an AS/400 repair specialist at over $100/hr are being floated.
Finally, I took a last ditch effort to fix the nic and managed to pull it off.
I let them know that I had managed to fix it and they all breathed a massive sigh, but before they could finish I let them know that this might be the last time it comes back up and that we need to back it up somewhere before that happens.
6 weeks later we had a nice shiny cloud based AS/400 and I held the funerary rites as I shut down that old lumbering beast for the last time.
Final clock, nearly 25 years of service.
How communication is sometimes bound to work.
Those who expect agents to perform all mental steps on their own to understand situations are in queue for harsh surprises.
Maybe they were talking about T1?
I’m getting very r/thathappened vibes.
- bugs that customers complain about.
When asked for why you can explain how the messy codebase is causing problems and how there no bypassing it.
I've seen this in a big medical equipment company where the CEO made a whole speech about how he now understands how legacy codebase is making our work very difficult, it caught me by surprise to see a C level bring up the topic.
Like some IT version of ‘the whippings will continue until morale improves’.
More realistically it was caused by those messages, but with a 4 year lag time, back in the days when the CTO decided to start the project from day 1 with a team of "senior" consultants that were all "full stack" and "excited to learn AWS".
However that doesn’t mean it will be invested in. Clean Vs messy are binary descriptors that devs love to throw around. A lot of time messy code is not worth fixing. Managers have to make judgement calls. But what they should always let you do is marginally improve the code each time you’re in it, and especially let you invest in test coverage.
Ultimately, trust is what governs how much oversight you should be expected to suffer in an organization. In my shop, I don't have to ask for permission to do anything unless it touches our customers in some way. If I see a machine that is struggling or a SaaS plan that is reaching quota, I simply upgrade. Every now and then someone will ask me about a new bill or increase, but I don't have to subject myself to multiple depositions to move the ball forward.
The suits should consider the downsides of creating an environment where one has to plead for permission to conduct the most trivial tech work. How much innovation is going straight into a burning trash can over convoluted change policies that were established in some ivory tower a decade+ ago?
Can we somehow reframe the organization in a way where the business is modeled as a customer of the IT team? That seems to me like a much simpler way to think of things, because at the end of the day if the business fails, the IT shop is has no raison d'etre.
The terms of this relationship (contract) should be approximately along lines of "you provide us a service with a certain level of quality and in return we grant you an annual budget of $X".
At a certain point, having detailed analysis of every last expense will cost you way more money than it will save.
There are different military philisophies, not all are top->down strict hierarchies. E.g. in the Prussian army of old, missions were communicated through objectives (I want to achieve X, your part is to do Y, while these other guys will do Z), and execution is up to to the local officers who actually see the things as they are locally and have all the knowledge about what they need to do, how it fits in the wider plans, and are best placed to decide how to do it. Furthermore, if an office or NCO disagreed with their direct commander's orders, they could go above their head to their commander's commander to protest. Add in the general staff system where staff officers had to service in command as well as part of their training, and could counteract commander's orders, it made for a very robust and flexible system capable of adapting to evolving situations and where officers didn't hesitate to argue with their superiors, go above their heads, and even disobey orders if they really thought it made sense.
This is called "mission-type tactics" or "mission command" (DE: Auftragstaktik): https://en.wikipedia.org/wiki/Mission-type_tactics
It became somewhat out of fashion at the Modern age, but AFAIK no army operates purely on "your job is to respect hierarchy decisions".
A significant part of why early company-owners in the US preferred slaves is not because their labour was free - they were expensive assets, particularly skilled ones - but because free workers were less compliant to hierarchical control. They could walk off the job, they expected their opinions to be listened to, and you couldn't whip them into compliance. There's basically a straight line from plantations through Gantt then Taylor and his Scientific Management to McKinsey's Budgetary Control.
Culturally northern corporations tend towards the West Point Peter Drucker management by objectives schools of thought.
Whereas the culturally southern plantation class is as you describe. Walmart is always my go to example.
This also roughly tracks geographically with strength of Labor. To no one's suprise, the neoliberal (confederate) generations long assault on Labor means the southern style has been waxing. Hopefully, with the youngs' (Gen Y & Z) new found enthuasism for Labor -- in the form of life, liberty, and pursuit of justice -- the slave owners grip on our society will begin to wane.
Try to think of it from leadership's perspective. For the sake of simplicity, let's say there are 100 similar initiatives proposed each year, each costing $1 million. That's $100 million per year in pure costs to the business (ignoring amortization, tax shenanigans, etc). Even for a bank, that's real money, to be strategically spent and not wasted.
With that setup, making leadership feel the pain and understand at a visceral level why this particular million dollars is well spent, is all at once a good strategy, rational for leadership, rational for IT, and a healthy-ish dynamic for the business.
Aside/tangent: In a very real sense, executives are managers of the finite time, employees, expenses, etc. of the business. Now, you could, and I would, argue that they're nearly always not especially good managers, and that the purely rational choice from "the company's" perspective is to pay them more like regular employees and not so lavishly. However, "the company" doesn't make decisions, people do, with all their political games, incentives and self-interests. Since executives control the flow of information/decisions/resources/money in the company, they siphon off an out-sized share for themselves, acting as a (perhaps inevitable?) parasite on the host company. Fixing this problem is left as an exercise to the reader.
The core of the idea is that the decisions that the overall body makes are so much worse if you try to combine managerial control with budgetary control that you want to separate them as much as possible. Set goals; provide budgets; don't link the two. Yes, as an executive you have responsibility for the overall spend. So employ people you can trust to do that well, and don't sweat the details.
Reminds me of the sociopaths in The Gervais Principle. Highly recommended.
Have confidence in your skills and the rest are details, especially if we talk about IT where job mobility is generally way above average, you really don't have to kiss assess or be worried what others think/say about you to be successful.
1. They are never at fault 2. It was always the fault of the prior leader after a re-org
It's perfect.
Or to put it other way you know that the 100% limit is a theoretical limit only achivable in ideal situations, so talk about the utilisable limit. People need to be fed the right information to be able to make informed decisions. And since it is IT who is paid to understand the tech things it is their job to communicate the peculiarities in a way the suits can understand.
Now maybe they have done their best job at this, but the article makes it sounds like they just sent a memo without this info.
For comparison imagine a bar decides to have 1 server based on average demand. Then Friday night comes.
This reminds me a lot about Covid perception by authorities: it's as if people in charge are all unable to make any kind of anticipation based on existing trend and have to see the disaster in front of them before reacting.
128 is one third of the way to 2 millions. And 2000 is more than half the way.
Having a vocal minority protesting against something usually don't slow down politicians when they have clear goals[1], but they become a convenient excuse not to act when they don't want to.
[1] I've marched in vain for many causes, most of which had much, much more intense protests than what Covid restrictions triggered.
With COVID, people who felt strongly about staying at home, were probably the minority, especially as time went on.
If a new pandemic starts in 10 years, people will still be burnt out from the last one and resist closure in greater numbers, immediately.
Yeah definitely as time went on. At the start I think a majority probably supported staying at home but after 3-6 months the numbers flipped.
The lockdown did save lives, but could not do anything else.
It's obviously something that governments who failed to act want you to believe, but it's not definitive truth. Maybe the uncontrolled situation in Iran would have been enough for the pandemic to happen, but travel from Iran is also much less important (in volume and economic consequences) than travel from the US, France or UK, so there's no certitude here.
And again, late lockdown did save lives, but with huge economic cost whereas early lockdowns saved many more lives with only a fraction of the economic cost, people who delayed the lockdown bear the entire responsibility of the additional deaths and economic disaster.
Creating calamity in a controlled way is a tried and true way of being sure one is prepared for the time it strikes uninvited, see DiRT exercises.
https://english.stackexchange.com/a/546162
>The Register itself uses the term Regomiser as a user name generator for its registration. If this contributor is a "reader Regomized as 'Felix,'" it would mean a reader registered and was reassigned that random name.
Instead we self-reinforce the image of childish kids doing mischief and pranks when the things don’t go our way.
The CTO was at fault in this story, for failing to convey the gravity and for being used as a doormat if things went bad.
Going for the neck would mean documenting everything and letting the bomb tick, and when it exploded make it clear to even higher-ups what was happening.
You'd probably go as well but that's not a good place to work anyway.
But it's that if there's anything executives are good at, it's looking at graphs, extrapolated trend lines, and setting deadlines before disaster strikes. This is basically one of the core, day-to-day competencies of the executive suite, because there are so many things that can go wrong regarding finances, infrastructure, legal, etc.
There are plenty of things executives can be bad at managing, e.g. when it comes to tech debt or user experience or company morale -- especially anything subjective or hard-to-measure like these.
But capacity planning for any measurable resource, be it an internet pipe or otherwise? This is MBA 101 stuff. So this particular story just reads like a fantasy of "tech people smart, business people dumb". If an executive team couldn't handle something this simple, there's no way they could have been running a functioning, established bank in the first place. It just doesn't add up.
If you know that installation of the new line will take N months, and traffic is creeping up X% each month, simple arithmetic will tell you how far ahead you should order more capacity.
The instructions of waiting until 100% shows a lack of understanding of logistical capacity planning (perhaps through a bad explanation/communication of the problem).
Like for example the business was struggling and they weren't expecting the same growth i.e. their capacity planning calculations were incorrect and there was no need for an additional link.
(It's possible that they did, and the retelling is so biased as to exclude that fact. But it seems unlikely.)
Most of them seem to barely get by by operating on the surface level strategy of "do whatever x creates y money more". When they succeed, it's their good job, when they fail, it's the bad job of everyone under them.
They only look at surface level money, they'll take surface level "we can make 35% more if we do x" rather than the deeper and more considered "we can make 25% more if we do x but limited by y for z reasons because x has a negative impact on customers but mitigated by y, costing 10% but guaranteeing the 25 via happy customers and stable systems".
Actually you can scale that out to more than just the finance departments. Another example is in large orgs, tech teams don't meet customers and don't understand real world usage and subsequently don't care or put thoughts into the implications of their actions.
Inefficiencies of scale? Would that be the term for it
Don't let the bean counters force budgets - incentivize them.
They are made of people.
They get too big.
Over time, the system had grown to be basically _THE_ source of truth for the release management process for a Top-500 website of the era. Go/No-go decisions were based on looking at the trend-lines for bug graphs. Projects were merged to "trunk" only if their P1 counts were low enough.
When I decided to leave I wanted to make sure things were left in a sustainable place, and asked around for who to transition the server to (it was a random beige box in the QA Lab, circa 2005). "It's Debian running MySQL, it has an auto-update script which will bug you if you need to update packages, here's how to restart the server, here's the root password, here's how to update/deploy it, docs are in this wiki."
I'd given 4 weeks notice, and week 2.5 rolls around and even though I'd kept asking around about who to transition this pet project to, nobody was swinging by my desk (so to speak) to pick up the responsibility.
So I made a little script: `@hourly: if (( $RANDOM % 2 )) ; then ifdown eth0 else ifup eth0; fi`
Suddenly people started showing up at my desk... "What's going on, can you fix it, what's happening?" Sorry, nope, can't fix it, the network on it is down... see, I can't ssh into it, by the way, this is what could very easily happen once I've left. Just wait an hour or two and it should come back online. If you send someone over who you want to be taking care of this machine I can walk over to the keyboard with them and show them how to fix it.
Once they committed to sending over someone to transition to, I turned down the BOFH to a 25% chance of 15 minute outages...and all was right with the world. Later on that week, a really good guy, one of the main end-users of the software (sorry Mike!) came over and explained the relative immaturity of my actions, but I couldn't help but point out their effectiveness.
In retrospect, it was an internal "API brownout" but that didn't have a name yet. The concept of "controlled unreliability" (eg: chaos-monkey) being a tool to increase business resiliency is something to take to heart, if even only unofficially.
Or learn to live with random 15m outages because it’s apparently less frustrating than doing something.
In retrospect, probably frightening how quickly the business could become accustomed to random 15m outages as a cost of doing business.
Imagine if politicians' https connections could be identified and automatically downgraded to http.
We know that such a system will inevitably lead to other groups also gaining the ability. Rich criminals, security services all around the globe (friendlies and the enemies too). Either by bribing, or impersonating authority figures or by stealing the backdoor keys. What is at dispute with the politicians is the probability of this happening.
How would a service voluntarily leaking the private information of politicians convince them that we cannot build a backdoor into encryption without that backdoor proliferating widely?
These people deserve to be fired and the behaviour is disgraceful. I can't imagine working with anyone like this.
They applied proactive network management to fairly allocate an expensive resource according to the anticipated needs of the company.
I say these people were skilled in saving the company’s ass. They could have just as well waited for the “I told you so” moment, which is what I learned to do.
I mostly keep the "I told you sos" in my head for these reasons.
But then I realized that IT abused their power and lied to the suits which is outright wrong. At the end of the day you’re in IT and the suits are in charge. Don’t like that? Find another job or grab a position that gets you this power.
The problem is: you don’t always oversee the consequences of your roque actions. In this case the stakes were low, but where do you draw the line? If I were a suit and found out that you deliberately throttled the connection to force us in a decision I would be furious. That is simply not your call.
They saw the car sliding towards the cliff and did something about it. Nothing that would cause immense damage, just a minor annoyance.
To skirt rules to get shit done, or watch everything burn and say I told you so. What's your choice?
I mean, you clearly have a paper trail showing you raised the issue at the highest levels twice. Not really sure how you could be blamed.
> To skirt rules to get shit done, or watch everything burn and say I told you so. What's your choice?
A couple of months of inconvenienced customers isn't exactly "watching everything burn", nor is anybody childishly saying "I told you so".
You're just respecting the choices management made. It's 100% their responsibility, not yours. It's not your job to save the world, or save management from their mistakes.
This isn't a situation of life or death, where you're in the military being told by your commanding officer to commit war crimes, where you have an ethical obligation to disobey. This is just going to inconvenience some internet banking customers. Your job is to raise the issue and let management deal with it and abide by their decision. Not to go around management.
This is sarcasm, right?
The "highest levels" can't possibly be wrong, so either it's space aliens, witches, or you.
People literally generate paper trails so that they can cover their ass. Many times after a particularly controversial meeting, I've immediately written out a summary of what was agreed upon and why, e-mailed it to everyone in attendance, and included the note to reply if anything is incorrect. Then when somebody's upset 4 months later and tries to blame you, you show them the e-mail and the lack of any disagreement.
Obviously at the end of the day, legally a company can fire you for mostly any reason at any time. But generally speaking in corporations, there are processes in place around penalizing employees that require documentable facts.
Of course I'm talking about places that have at least a couple hundred employees, with an HR department as such. (Smaller companies are much more at the whim of their founders, for good or bad.) But this bank certainly sounds large enough.
Choosing not to undermine them is not bootlicking. It's called being a professional.
Going behind their back because you don't agree with their decisions or actions is not professional.
Here's a quite common problem. Execs tell you to listen to the customer, that's where the money comes from. Your direct line manager cares exclusively about himself at the customers' cost. What do you do? Which level of the hierarchy would you obey?
> Going behind their back because you don't agree with their decisions or actions is not professional.
Should you go behind the back of your manager and report his corrupt behavior to their manager?
Please don't take this as snark, I've been in this situation multiple times and it's actually a difficult moral decision depending on the stakes.