Comment by weitendorf | Hacker News Reader

Comment by weitendorf | Hacker News Reader

Parent Full thread

Definitely agree that domain experts are preferable to domain owners, and that overly explicit specialization causes problems, but at the same time I think it's quite possible to stray too far in this direction too, and that the author didn't really address that.

The problem is not just that the Designer might Break the Build or ship something broken one time (but know how to fix it). The problem is that stuff can (obviously, not in all cases, and not something you should just assume will happen without good evidence/reasoning) start breaking all the time because you have too many people changing things that interact with other things they only partially understand. Or it's that one team/"domain" ends up downstream or dependent on another, and locally optimal but globally suboptimal decisions made upstream (eg a bad but quick to implement data model, or adding more dependencies on something you're in the process of getting rid of) cause problems downstream. In these situations your "egoless domain experts" might actually need the authority to force these things to stop, or might get burnt out spending all their time firefighting or playing hero.

There are also some kinds of engineering mistakes that are literally business-killing, like major security breaches/data loss. Mandatory code review isn't a "feel-bad program", it's precisely to guard against disasters like this (also I'm pretty sure it's literally required by SOX/SOC2 and I'd certainly want my software vendors to implement this).

That's why I think this is actually a balancing exercise that is highly case dependent, not something you can merely say "just empower people we're all on the same team". If your team is highly skilled, conscientious, and motivated you can give people a lot of autonomy - if they're mostly unskilled and don't give a fuck about the quality of their work, you can't give them much autonomy. If the scope of what you're working on is huge (so huge that any one person can only have a working understanding of a small part of the whole) you probably do need more process and guardails in place than a smaller project.

> In these situations your "egoless domain experts" might actually need the authority to force these things to stop, or might get burnt out spending all their time firefighting or playing hero.

The broader approach here is to encourage frameworks where changes must go through policy-as-code frameworks. SMEs write policy that is then enforced (or just warned, for some period ahead of enforcement) by the policy framework. Policies can encode exceptions, but changes to policy (i.e. to add exceptions) require SME review. The benefit is - stuff that passes policy doesn't require explicit review, so in the normative case autonomous teams are not slowed down, and when they are (due to policy failure), then it is for Good Reasons.

> Mandatory code review isn't a "feel-bad program", it's precisely to guard against disasters like this (also I'm pretty sure it's literally required by SOX/SOC2 and I'd certainly want my software vendors to implement this).

High-trust environments trust engineers to understand the difference between "this is a serious change so I need someone to review it" and "I'm just fixing a typo here so I'm going to rubber-stamp it." But yes, SOC2 requires mandatory code review, tickets, the lot. Yes, it slows things down, but that is the price you pay to become a Serious Enterprise Vendor. No, it doesn't fundamentally erode high-trust culture. People can just mark their MRs as "I need a rubber stamp" and find someone else to rubber-stamp them.

If you care about the experience of your co-workers, you won't repeatedly break stuff. (Or if you do, it's because your tooling hasn't scaled enough to keep up; you need a staging area or precommit hooks or whatever.) If you know you're trusted to do your job, you will work to maintain that trust.

But things need to be architected for trust. If you have lots of rigid processes, then there's no use for manually executed tools that will only be used if someone decides to use them. People will lean on the process, and every time something goes wrong, the process will be "improved" (as in, catch one more potential issue, at the cost of taking longer and catching two more non-issues and making people batch up their changes more, causing more conflicts, etc.) In a high-trust environment, people will use the manual tools when appropriate, and so there's incentive for architecting things such that the results of those tools and processes is useful. Tests are more likely to test real things. Staging environments will be made to better reflect prod.

If people care about it, they'll make it work. If they don't, they'll make it "work".

But yes, I agree that as you scale up, more and more things will get missed and you'll need to balance things out. It's just so, so common to go too far in the rigid low-trust direction.

People are so terrified that something might go wrong that they'll do things that end up making sure that something will go wrong. It'll be something later, or somebody else's problem, or whatever.

Yeah you really can't go this hands-off on deployments when the stakes are high. If someone could be physically harmed by bad code, then it all needs to be reviewed. If you can be hacked by a rouge deployment, then it all needs to be reviewed.

And more generally, it is not fun using a lot of mainstream software these days because it is a buggy mess, and I believe a lot of that dysfunction is partially related to the low standards for correctness that development teams have.

Reply on news.ycombinator.com