Hacker News Reader

Comment by im_down_w_otp | Hacker News Reader

You're going to be very sad to discover the sentiments around software-intensive systems testing in a lot of places & industries that you'd rightfully assume should know better and do better.

I had a safety critical embedded engineer who had worked on everything from weapons systems to automobiles first not have any idea what property-based testing was and then proceed to tell me at length how it was pointless because unit testing is all you need as long as you keep making your units smaller and smaller.

At the time I worked in FinTech and was aghast at what I was hearing as they worked on self-driving. If I had to pin down what made me shift my career path into "cyber-physical systems", it was probably that conversation.

Since making the transition and later founding a company dedicated to testing in that domain, I've heard dozens and dozens of sentiments even more concerning.

View on HN

> and then proceed to tell me at length how it was pointless because unit testing is all you need as long as you keep making your units smaller and smaller.

I've encountered devs that think this nonsense before, but always assumed that this attitude couldn't be held with systems that are actually critical. I'm a bit less naive now.

I thought that too, early on. Sort of a perfect-bricks-make-perfect-walls idea. And when problems emerge at the integration stage, it’s often traceable to a gap that a lower-level test could have caught. So the mentality is a bit self-reinforcing.

Exactly. All the products we make are focused at the integration and system level because it's crystal clear that's where enormous amounts of issues arise and unbounded numbers of 11th hour fire drills occur. The various industries who've had multiple product cycles with lots of software are starting to figure this out, but it took awhile.

Despite how obvious it is with a little outside observer perspective, the "perfect bricks" way of thinking is pervasive because that's how the builders and supply chains are organized.

Which is silly, because the whole point of integration testing is to show your bricks don't fit together as expected.

Or that your bricks aren't square, or they are actually made of sand, or that they collapse under load.

To leave the analogy behind, any non-trivial component has a testable surface area, but typically has additional modes of behavior associated with internal state, environmental conditions, or other areas that well-meaning unit test writers didn't think about.

I have often found issues in simple caller/callee pairs of two components, both of which are tested, but the caller contains subtle expectations of the callee that the unit tests don't match up with.

Software testing needs a unique mindset compared to traditional hardware reliability testing. The software doesn't "wear out" like the hardware but tends to fail in coordinating functions. Too often, we rely on simplistic measures like a hard restart to manage coordination failures, but that isn't always an option on safety-critical applications.

Taking that argument to the extreme, why test any new software at all when the underlying instructions have all been thoroughly tested!

Well we do that all day long when we write to a file system and expect it to actually persist data to disk. It's usually only db engines and loggers that attempt to flush, and even then the disk will usually lie in response.

Ariane V is an interesting case study in that regard [1]. It failed because they didn't feel the need to retest the software reused from the Ariane IV design. No need to re-test the bricks that already worked before...

[1] https://en.wikipedia.org/wiki/Ariane_flight_V88

Reply on news.ycombinator.com