Hacker News Reader

So I worked for a company that had loads of process for even the smallest changes. It was frustrating, but what I came to realize was that the company was not interested in fixing bugs or building features - they were primarily interested in not breaking things/keeping the status quo. This was an established business that was concerned with keeping the business running, not chasing the latest and greatest. Perhaps a lot of it was just CYA, but also means it was pretty hard to "move fast and break things".

That sort of business sounds like what we want in milsec and government suppliers, no?

Reliable even down to the bug behaviors...

Not only. Also some software for financial sector, for suppliers, for maintenance too. Basically, anything that do not crash a system could be a feature. And sometime even have logical and good reason to exists.

The reliable bug behavior is an underappreciated point and can be hard to understand. I work on software that gets deployed on-prem by customers and had very powerful capabilities for deploying your own code to it etc. There have been a few discussions about some bug fixes and in particular security fixes being appropriate in a patch release when following SemVer. Something might be unintentional or insecure, but customers might be building on the broken behavior and now deploying a patch breaks their apps. The line can become quite blurry.

This sounds like more a deployment/release process problem than anything else. If the customers are essentially deploying untested code straight to production, they're doing something wrong. It might not be their fault, maybe they don't know they should have a staging environment first, or maybe your pricing model makes it prohibitively expensive. It's not as simple as "your customers are doing it wrong," but if you upgrading a package breaks their production environment, someone somewhere is - or more likely, many people are - definitely doing something wrong.

It is not necessarily about breaking the production system, just breaking the system because it depends on some bug or contains workarounds that break if the bug is gone. Even if it breaks before production it is still broken and might - depending on how old or central the bug was and how much code was build on top or around it - require substantial effort to fix.

100% agree it just sounded like in this instance the first time they'd hear about it breaking a customer's workflow[0] would be in that customer's production environment. I've been on both ends of solutions where customers are either technically or financially incapable of having robust test environments for even critical software, especially if they're relying on someone else to write it.

[0] https://xkcd.com/1172/

Yes this happens all the time, I’ve seen it everywhere I’ve every been an engineer. Bug fix roll back due to causing an outage because it broke the defacto API being depended on.

I have had this concern raised, and I'm unclear how, if you adhere to that model of compatibility, that you don't effectively dispense with semver altogether and make every release a major version bump, as every change is breaking if you relied on broken or insecure behavior in earlier versions.

IMO it makes more sense to communicate the expected impact of the changes, so that downstream can read a patch bump as unlikely to cause issues.

Yeah, it gets fuzzy and loses clear meaning. Unfortunately, major releases are expensive in that support contacts are typically guarantee support durations for major or minor releases. Having dozens of versions that all require separate patches quickly can absorb much of your orgs productivity.

I imagine it depends on the specifics. Closest I got[0] to the defence industry was all the anecdotes my dad had about Plessey and GEC Marconi.

I'm sure it's important to have a stable technology to train on in the military; at the same time, I remember one of their open-days they were showing off a civilian in-flight entertainment system that was so clearly overpriced for what it was that I knew even as I saw it that I could've done better for lower unit-cost. This isn't to show off, even despite my having not yet finished secondary school at the time — my solution would've been little more than "glue laptop to chair/wall in front", as tablets weren't something people discussed much around me in the 90s — it's just that the business solution was really unimpressive.

I assume there's secret military equivalents to that public project.

[0] if you exclude the job interview at Lockheed Martin for the year in industry paid work experience part of my degree, which didn't go further than an interview

If they just buy whatever is most cost affordable then the risk factor for trojans goes sky high. Even with major brands this would be a significant issue (see HDD manufacturers bs with drives shipping with malware in the firmware).

Theres a reason there's so many steps to accept hardware.

No, of course not, if your military doesn’t develop tech faster than the competition then you lose.

I too worked for Google once.

That kind of slow and steady approach actually sounds pretty reasonable for some cases. Sure it won't win the market for a trendy new consumer app, but a lot of software is basically feature complete and should prioritize stability over whatever's currently cool and exciting.

I have an old account with a bank that has hardly changed its website design in well over a decade. It's great! The information is clearly laid out in tables and everything loads super fast. Every other financial web app I'm familiar with has a much more "modern" design and is categorically worse to use. One site literally takes over a dozen clicks to get to basic information and evey single page load is slooooow.

Bank webpages don't need to be more than HTML tables, with perhaps a bit of JS thrown in to ease element collapsing, etc.

I worked on a radar system that could see precipitation- think doppler radar map view. The ability to do this was in the contract but there was no way to view it. We had Matlab scripts so we the devs could view it and the weather people wanted them. We weren't allowed to hand them over without a contract. They'd pop in multiple times a day to view our doppler.

The adversarial nature that defense companies treat our military is appalling. I'm convinced that refactoring a lot of this BS could get us way more with less $

If the viewing software was provided, the support would be included and that will cost money in the long term. That is something that the company didn't agree to provide nor the government to pay for. In this case, the adversary wasn't the contractor, it was the users in the government and the government employees holding the purse strings. No contractor will turn down the chance to extend the contract.

A lot of this adversarial nature comes down to how projects are awarded. Government employees have very little discretion to choose projects, so they always go with the cheapest bid. But no large software project ever has had 0 change orders.

So the only profitable way to win these contracts is offer the cheapest bid by delivering as little as possible while being in spec, and then pillaging on the change orders.

What about the "new defense" companies, who use VC money to build demos? Surely they eventually want to build moats and sell razor blades, but for now what they're doing seems different?

I have encountered a situation a few times where a bug that went unnoticed for a while was actually keeping another, more serious bug from surfacing. So when the first bug was fixed, it caused more serious problems elsewhere in the system. I could see where they're coming from where any kind of "fix" needs to trigger a careful reevaluation of the whole system. That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.

> That being said, I don't see a need for the Kafkaesque months of planning and budgeting that follow.

It's a method for transferring billions of dollars from public tax revenues to private companies which are cozy with the state. It has nothing to do with safety or reliability.

All other methods of stealing tax money are illegal.

Would be nice if we spent some time cracking down on that kind of tax theft rather than wasting time on drag shows and the 3 dozen teens in the whole country on puberty blockers

you got it mon ami. that's the whole point of making a kerfuffle about drag shows. or gay marriage. or even abortion.

ain't no one talking about tax cuts, or getting after the rich, cuz of that.

I’ve also encountered that a few times where a fairly anodyne bug in a codepath prevents a serious security bug from being reachable. With my attacker hat on it is very tempting to just report the first one…

My experience has been that you can use open source software, but security considers that to have a high risk and you need to make a business case for it.

What isn’t a security risk? A closed source vendor supplied application that has its headquarters in the US.

You can sometimes find pockets of innovation in the defense sector, but it’s mostly filled with bureaucrats with engineer in their title.

To be fair, securing the OSS supply chain is a tough task, especially when pulling in new versions.

And yet, things like the Patriot missile system get built. It's pretty wild what an 800 billion dollar budget will paper over.

Modern defense contracting has become, first and foremost, a process to transfer government funds to private hands. The fact that any useful products or services happen is purely incidental.

you can say that again

It’s a broken system for sure…

HOWEVER when your acquisition system is designed to ensure that a nuclear detonator and a LAMP application go through the same process, these kinds of “overkill” or “nonsense” requirement responses happen.

We’re long overdue for fixing the FAR and federal acquisition process, which is the root problem. The legal system and processes built over the last 50 years, and as a result the people inside it, doesn’t allow for simple and efficient differentiation between projects legally so you end up with the same hammer meets mail approach.

Part of this stems from government accountability issues. If you fix that bug, and you don't go through the process, no you didn't. At any time the government can ask for a contract performance review, and doing things the easy way means lots of things that should be tracked don't get tracked... which means they never make it into the contract performance review.

They mean well, and I understand how hard it can be for government agents to hold massive contractors accountable, but the purpose of a system is what it does... so we wind up with entrenched bureaucracy, and the players who don't subscribe to these approaches wind up getting outcompeted because their on-paper performance doesn't measure up. Even if they're in fact a million times easier to work with and do a great job.

But you'd be just as upset as his employer if you'd come across a binary release built from one commit ahead the GitHub repo. The absurdity is in the amount of overhead, not the fact that there are overheads at all.

> A couple of weeks later he comes back, and reverses the bug. He had told his manager about the bug fix, and was berated for fixing in. In order to fix that bug, it had to be reported correctly, go through software update meetings, go through budgets, and the whole nine yards.

What do you mean you went and just fixed it!?

We need all of middle management to charge for the fix!

s/defense/regulated industry/ I found a bug in my company’s software product once. The code was using the user’s display name in a lookup rather than the username. I grabbed a QA engineer, he agreed it was a bug, and he put the wheels in motion. I don’t think that fix ever made it into a release.

Our software was part of the stack that our regulated customers were using to document their compliance. Every year or two, they would audit our development process to insure that untoward code wasn’t slipping into the release. Getting the fix into a release would have required that the bug be prioritized, fixed and qa’ed. Never became important enough. It burned me at first, but I’m not sure it was the wrong outcome.

Do you have an opinion whether this kafkaesk bureaucratism is especially well established in the domain of software or is it a broader pattern? Where does this come from?

Remember: Regulations are written in blood.