The simplest response for both is to delegate: "<person y> now has temporary or permanent authority in the decisions <person x> used to have". That means you need to have a person y for each person x and there needs to be enough trust that person y will make decisions that x would approve of (even if sometimes different). If you're a single-person startup, there is no person y, but then these questions aren't really relevant. As the Pinboard FAQ says, the answer to what happens is the bus will be fine.
The regulatory demands in financial technology, and especially in those industries where both the regulators as well as majority of the players may not be technically literate, are effectively the anti-thesis of robust and reliable engineering. When the industry regulations demand that for every potentially disruptive action (recovering from an outage is very much potentially disruptive) there has to be a named individual - not even a role, but an individual - to sign off on every single release step... This is why an outage can last for hours. An engineer, or even a lead, is with high probability not allowed to sign off on an out-of-hours deployment. The fix might take 20 minutes to identify, 5 minutes to implement, 15 minutes to test, and 10 minutes to roll out. But until the organisation can get the named [supposedly responsible] person online, the fix must wait, unreleased.
And the irony? The wider dysfunction may be imposed by the regulations, but it has been requested by the industry at large. It is the result of the industry players, majority of whom are incapable (read: technologically illiterate) of interpreting prescriptive regulations. So instead of figuring out how to meet expectations and adapt their processes, they have been lobbying for the regulators to come up with highly descriptive playbooks and step-by-step instructions. These sequences then specify rules and requirements that allow technologically incompetent players to comply, even if they do not understand why they are doing the things.
And then these regulatory playbooks become the One True Way[tm], effectively forcing dysfunctional practices, but also actively preventing any process improvements.
It's cargo-culting taken to the extreme: "do this and you will not be found to be non-compliant, even if the quality is utter shite". The worst part of this all is that the descriptive regulations make every effort to strip engineers of their agency or decision power. Every "modern" best practice is explicitly ruled as non-compliant and hence effectively illegal. I use scare quotes around modern to highlight that these practices are by no means modern and have been known since ancient times. There are research papers from 1950's that explicitly endorse [rapid] iterative bottom-up approach as the only sensible way forward.
The finance industry regulations explicitly reject them because these studies assume that engineers can be trusted to know what they are doing.
So... to anyone wondering why finance industry jobs are generally thought of as soul-sucking: that's why. They are subject to regulations, expressly designed to deprive individuals of their agency.
The desired result of the regulations is, first and foremost, accountability. This includes, but is not limited to, audit trails and provenance. But because the regulations have been written to specify exactly One True Way[tm] of reaching an adequate level of accountability, they end up causing nasty second-order effects.
As sibling comment by PeterStuer mentions, the unfortunate end result is inventive interpretation to avoid the worst of the disruptions. So from my point of view, the rules put in place to _enforce_ accountability have created perverse incentives to _reduce_ proper accountability.
That can not be a healthy situation either in short or long term.
Interested in this... any links?
The most valuable sources have been just 5 (yes, five) research papers / reports. Luckily they sport citations, but sadly I haven't poured over more than a fraction of them so far.
DOI: 10.1109/MC.2003.1204375 ; "Iterative and Incremental Development: A Brief History"
DOI: 10.1109/AGILE.2013.17 ; "Continuous Delivery? Easy! Just Change Everything (Well, Maybe It Is Not That Easy)"
DOI (with link): https://doi.org/10.1002/spip.344 ; "The agile professional culture: A source of agile quality" [the article itself is pretty light on useful material but it contains some of the strongest individual arguments I've seen, all in one package]
Direct PDF link: https://www.futureworksconsulting.com/perch/resources/pdfs/t... [the valuable material is on the first page]
DOI: 10.1.1.428.5843 ; "Evaluation of Quality AssuranceFactorsinAgile Methodologies"
---
There are plenty of other reports and papers on the topics, but they bring very little to the table on their own. These five have provided me with a decent starting point. The references to findings and reports from 1950's are in the first one, btw.
You're right I haven't been directly exposed to such regulations (our emergency releases also require sign-off from management but there are multiple people who can approve and different people for different stages) and I don't ever intend to be. Even for something much lighter. At my current job I refused a request from my boss/boss's boss to go through the process of obtaining a Public Trust Position which would be necessary if I needed to debug my team's product for one of our government customers. Fortunately I didn't have to quit over it, and I still got promoted later. Other people on other teams just went along with it. So I guess I'd amend my thinking (aided by your comments on lots of the regulation in finance being self-requested) that for people who get themselves into a position where it's "not possible" to go 100% on vacation, regardless of the level of dysfunction at the org, many of them generally don't mind it. If they did, they could switch jobs, and anyone concerned about accepting such a deal should weigh that instead of imagining that there's nothing to be done.
If fecal matter hits the rotating turbines often that you are never 100% on vacation, then you have fecal hitting around all the time. It is not like seniors would be on vacation 50% time of year.
> Often times, theres an issue with this or that feature in production
You should not have often issues with features in production. You should have them once in a while, rarely. The situation that seniors are always desperately needed to fix crisis during their vacation requires workplace that is having crisis pretty much all the time.
Capable management does not generate projects that are in crisis all the time.