Ask HN: How do you keep big software systems bug free | Hacker News Reader

Ask HN: How do you keep big software systems bug free

3 pointsakkad336 comments

We work on a highly complex domain specific software product in my company. Ours is a growing platform used for mathematic computations with many interlinked modules and new features being developed every day. Over the last months we have developed many new features and old code is used and modified in the process. We have found it hard to implement a failsafe system that ensures system correctness. It has been hard to ensure new features developed are correct and bug free and to ensure we do not break older features. The systems we have in place for catching bugs are human overseen and automatic end to end tests, integration tests in the code and unit tests. Unfortunately because of the sheer explosion of edge cases it is impossible to have a test for every possible edge case. Furthermore I feel our specs are transmitted in an ad-hoc manner through Jira tickets and it is easy to miss something or have some important detail get lost in the detail. I have been thinking of the feasibility of maintaining specs in a formal software language or using systems like Prolog to automatically generate edge cases to ease testing. I have no clue if this is feasible. How are such problems commonly handled in your field?

"Unfortunately because of the sheer explosion of edge cases it is impossible to have a test for every possible edge case"

You just can't keep any system bug free, the only thing you can do is improve your detection systems, codebase, and documentation and over time bugs will pop up less and less

Everyone's tips will be different, but I'll throw out a few:

* Make illegal states unrepresentable

* pure, static functions wherever possible

* use test generators (a la quick check)

* if you can't unit test important code, try refactoring until you can, rather than immediately reaching for integration tests and mocks.

I don't think you'll be able to model your system well enough to be able to run it through a formal language (I'm thinking about TLA+). But Leslie Lamport did say something about "the algorithm is not the code". That is, if you have any tricky algorithms that you invented for this project, you can formally prove those algorithms in TLA+ and then translate them to code.

We have found unit testing not very useful because ultimately it's not a single component that generates errors but the interaction between many of them. Also the language we code in makes the first two things very difficult to enforce

>> if you can't unit test important code, try refactoring until you can

> unit testing not very useful because ultimately it's not a single component that generates errors but the interaction between many of them

devil's advocate: what's are the costs to the users / customer / business if defects are shipped? how does that compare to the value of rapidly adding new features? if the cost of shipping defects is much lower than the value of shipping new features, and it costs a lot of money to change the process or tooling or product to reduce the defect rate, then one approach could be to not attempt to further reduce the defect rate and accept the current rate that defects get shipped.

in some contexts a high (or even low) defect rate is completely unacceptable, the cost of a major defect can be ruinous to users / customers / the business --- e.g. if defects in the product can kill large numbers of people, or if defects would cause a giant batch of cash trapped in manufactured hardware products to be recalled, or if defects could cause the company to lose money very very quickly (c.f. the knight capital automated trading feature flag thing). but many contexts aren't like this at all.

> It has been hard to ensure new features developed are correct and bug free

Count yourself lucky. Most teams find it impossible! :)

Reply on news.ycombinator.com