Hacker News Reader

Most orgs don't do 4-5 because they don't receive enough executive support to make meaningful organizational change. Sometimes the executives are clueless, sometimes they're absent, sometimes they're fighting budget battles against other executives and plotting high-level political plots with other executives and board members.

Surprisingly few organizations genuinely care about product or process. The market opportunities for companies which do, are staggering.

Can you talk more about the opportunities and how dev process is related to product and where the opportunity lies? Sound like a provacative yet vacuous statement to me?

Basic formula: 0) All companies are now software companies. See https://blogs.thomsonreuters.com/answerson/all-companies-are... etc. 1) Pick a well-established, non-high-tech industry, with poor technological adoption, which is not being dominated by a FAANG. Many options exist: banking, industry, food service, construction, transport, hotels, etc. 2) Start a small, local competitor. Differentiate through better technical service, using better dev process to respond quicker to market conditions. 3) Use that technical differentiation to ease scaling and enable growth. Ad infinitum.

Many enterprises are still stuck in effectively yearly release cycles, not to mention the number of enterprises still encumbered by legacy systems e.g. mainframes etc. Pick a "target" by making some phone calls, doing some networking, find a business which is encumbered this way, and start a competitor. It should not be difficult to steal business from a competitor which a) will take a year to release a response b) whose response will be handicapped by needing to support internal legacy systems with its response.

If you think easy targets like that don't exist, you need to get out of SV and spend some time with the rest of the world.

> 0) All companies are now software companies. See https://blogs.thomsonreuters.com/answerson/all-companies-are...

Technology != software.

This is a distinction that I've found the vast majority of software people implicitly assume doesn't exist.

A sibling comment points out that physical logistics sometimes can't be solved (competitively) by technology alone, but I posit that this is true even within technology. That is, not all technology problems can be (best/competitively) solved by software alone.

None of FAANMG depends exclusively on software, running on someone else's hardware and network.

G famously customizes their servers, and they had hardware frugality as part of their core strategy from the very start.

> encumbered by legacy systems e.g. mainframes

That may not be as great an encumbrance as you imagine, considering the kind of performance modern mainframes can deliver.

Are they over-paying for that performance, compared to best-case commodity hardware? Of course, since mainframes (for the latest models) are a monopoly. They may also be over-paying for the software and/or developers.

A startup with AWS-based infrastructure will be over-paying, too. Cloud may not be a monopoly, but it's at least arguable if the competition is robust.

That distributed computing system, with all the inefficiency added by the distribution, may be remarkably copetitive for someone that's paying below commodity prices for infrastructure. For someone paying 10x commodity to a cloud provider, not so much.

Sometimes there's side considerations which are technically simple but logistically hard to work around. Technical differentiation is only one way to gain a competitive advantage, but it's also not necessarily the most important advantage. For example, it would be incredibly difficult to compete with Walmart unless you had a network large enough to compete with its effects on the entire production chain. Only someone else like Amazon can compete in that space, and it's not because they're technically superior, it's because they're friggin' huge.

> 4) perform a postmortem to identify the next steps

It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.

Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.

In my experience, it's rare that any blame is sufficiently concentrated as to make assigning it sensible. A failure by a human is more commonly a failure in the process they were following, or in the supervision they should have received.

And there's a big difference between "blame-free" and "unaccountable". It's not the fault of a team of developers if they aren't experienced enough to take responsibility for a critical production service, but the solution is not to berate them for being inexperienced: it's to give them the support they should (in hindsight) have had in the first place.

Sometimes that is translated as “a lack of training” which is often a euphemism for someone did something stupid and we have to put in place reactionary consequences (training on what not to do) if they or someone else does it again.

I typically push back on conclusions that only involve trying to change people and don't also put technical safeguards in place.

There are definitely things you can't fully automate, but the goal should be that doing the right thing is really easy and making mistakes is hard.

Lack of experience is a nice word for incompetent.

It remind me of an outage a few months ago. Lost a pair of systems, couldn't login and run anything anymore.

Turns out, an operator ran a "rm -rf /home/user".

He was following a procedure that includes cleaning a subdirectory. The subdirectory didn't exist so he figured out let's clean whatever top path exists!

> Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task.

It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance.

However, I think the process approach is applicable to a much larger domain than many people give it credit.

One of the greatest software development quotes I've read comes from DJB [0]:

> For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits.

The difference in security bugs found in qmail and sendmail speaks for itself.

This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur.

[0] https://cr.yp.to/qmail/qmailsec-20071101.pdf

Yep. In many ways the "competency" of a person or group comes down to whether there was a designated person whose job it was to ensure they were following good processes. DJB does it himself, but most people don't do it for themselves.

There's "blame" and there's "fault". Faults are correctable, blame tries to place responsibility on actors. Few faults can be corrected without acknowledging where responsibility lies. There are multiple kinds of responsibility however. Responsibility for the cause and responsibility for fixing it. The same party who caused the issue isn't necessarily qualified (or authorized) to address it. The idea behind blameless postmortems is to be descriptive without calling out bad action, and evaluating the fault holistically, and acknowledging the reason most faults occur is a failure of processes not necessarily a failure of a person. Fixing processes is much more scalable.

Sure, sometimes someone made a mistake. But they're not to blame; the process was set up that someone could make a mistake. Change the process so if/when a mistake happens, it doesn't cause a problem.

> Sometimes a root cause is simply that a person or group was not competent in their execution of a task.

If an incompetent person is able to take down a competent person's system, the "competent" person is to blame.

In my experience that's quite rare, what is likely to happen is any postmortem will be used as a political stick to beat someone with.

> They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again.

This doesn't work. You shouldn't focus too much on making causes of bugs to never happen again, they will happen and they don't matter for operations as new code will bring back old causes of bugs and introduce new ones. What you should focus on is making sure that when bugs do happen they don't cause any trouble. Be realistic about bugs, expect bugs.

Of course it helps to make your application resilient to failure, but it does not help to ignore the causes of bugs.

I would say most bugs are entropy, or a human doing something wrong, or processes that haven't accounted for some edge case. A lot of these can be prevented, and a common way to do that is to watch them happen, document them, and then implement a mechanism to fix the cause when it starts to happen.

It's not a panacea but there's a lot you can do that does work. Maybe not 100% of the time, maybe not even 90% of the time for some bugs, but stuff that works nonetheless. Better unit test coverage, regression tests, new linters and static analysis passes, fuzzers, explosive runtime checks that catch problems earlier and more obvious ways, ...

sentry.io

The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems

This strikes me as extremely arrogant and naive way of thinking. "Customers, stakeholders and investors. Your interests have been damaged but rest assured lessons have been learned. We will continue with this approach as it makes sense to is"

not developing everything correctly is about caring about not damaging interest of people.

Having data loss while migrating database to new release version is never acceptable. Db migrations should always work correctly, no discussion.

Having some css issues where some menu is displayed in a crooked way is perfectly acceptable.

The attraction of "Move fast and break stuff" is in its simplicity and focus.

Once you start qualifying it with "it's only for cosmetic changes not for serious stuff" it stops being worthy of a discussion.

I bet this blog post was about javascript :p

About your steps. The problem I often see around work is that A bug in prod doesn't get patched in the mainline then pushed all the way up ASAP so it can't accidentally promote up again as it exists somewhere in test or preprod as well.

I like the CI/CD approach of just having everything go down the same pipe as it seems to be a decent way to at address point 5.

"I bet this blog post was about javascript :p"

You should bet more often ;). That's quite correct. Mostly Javascript with a bit of python.

I would add, write automated tests to identify that the issue was fixed if it is possible.

> Bugs will always happen in production.

This is not some universal law. It’s true only when the project/team is not willing to make the trade-offs (like development speed) needed to deploy bug-free code. For most software it’s better to release with bugs, but not so for all software. For some software, like safety critical projects, things launched into space, etc, it’s better to take your time and release without bugs.

Besides, good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs from reaching production. Sadly, this idea that bugs in production are inevitable is pervasive in industry.

In order for that to happen, you would need perfect practices, perfect requirements, perfect testing and perfect ‘change control’, and an inhuman ability to never make a human mistake in the understanding of any one of those things within your team of equally perfect people, so add perfect communication on to the list.

Anything less than perfection at any stage will be a vector for bugs, whether that bug is a syntax error, unexpected behavior, or a fully tested and functional feature that doesn’t meet expectation.

Mistakes are inevitable and not necessarily bad when you handle them well (move fast and break things says nothing about what you do with all the breakage, so feels immature). Calling them bugs doesn’t change that.

Can you point to a violation of this observation? Operating systems, CPUs, compilers, and software at large have all had bugs. Can you point to anything meaningful that has not been susceptible to bugs in production?

> good development practices, good requirements, good change control, and good testing can prevent the vast majority of bugs

prevention of the vast majority of bugs does not negate the claim that bugs will always happen in production.

I am skeptical deploying a non trivial project to production without bugs is a viable strategy, even for safety critical projects. I imagine the bugs are just much less severe.

It is not even easy to tell whether a piece of software has no bugs or just none you have seen.

Arguably some of the most conservative and careful devs in the world managed to crash a spaceship into Mars and you know they fully subscribed to "take your time and release without bugs."