Hacker News Reader

Over the years, I've run across this type of concern when authoring test suites which validate external system interaction, not just property-based ones.

The situation you describe appears to be ideal for having test suites which exercise the third-party API in ways the system your team expects to use it, especially if those suites are viewed as executable documentation. Even if others in the organization do not value automated testing (and, yes, this has happened a lot more than I care to remember), time spent on them bring value by having a reference in-house. Key here, IMHO, is regarding these test suites as verification of what the third-party claims in their documentation (if any).

It's important to limit time spent on making these tests, as one wants to avoid making a brittle suite which causes non-trivial maintenance costs for the organization. IOW, try not to go too deep down the rabbit hole.

> I just sometimes wonder if I write tests too often.

I try to do an extremely rough guesstimate as to the cost of not testing:

- Debugging sessions to reproduce latent bugs - Debugging sessions to track down refactoring mistakes - More thorough code reviews because you can't trust the testing suite to catch problems - QA time to catch/reproduce bugs - Costs of downtime when the bugs take out a critical system - End user, customer support, and/or I.T. staff overhead in dealing with the damage caused by uncaught bugs, or working around unsolved bugs

My usual conclusion is that nobody I work with (nor myself) does enough automated testing.

> Should you write property based tests of a third party custom api if you are on a short term contract and pretty sure that no one is going to maintain the testing suite?

That sounds like you're doing their work for them. If I'm accepting risk in a contract, my major assumption would be the third party libraries I'm using are fit for purpose. You surely can't be held liable if the other contractors aren't doing their work to an acceptable standard.

It's not quite that simple. Much of the problems of Windows reliability was from 3rd party drivers and apps (esp killer apps). They couldn't ditch the hardware or app vendors because their customers demanded them. So, Windows source was littered with all kinds of provisions to basically work around all that. They eventually forced driver analysis and safer language onto the ecosystem when that was too much. But a key part of their path to dominance and backward compatibility was working around known issues in 3rd party code.

Avoid this when possible obviously. However, it's sometimes a good idea when it's for tie-in to ecosystem that has benefits worth it.

Maintaining Windows is a very complex and special case though. I mean for most contract work you can safely assume the libraries you're going to be using are mostly bug free.

"Safely assume" is an oxymoron. Assumptions are a form of risk taking. The upside reward is that you save the cost of testing (and thus eliminating) your assumption. The (potential) downside is the complete failure of everything resting on that assumption. In many cases, the upside is small and knowable, the downside is hard to predict or contain - i.e. the worst kind of asymetric risk. Thus as a rule of thumb, assumptions are very bad practice, and should always be called out as liabilities in contract terms - e.g. "The cost of repair or rework due to any bugs found in the provided libraries shall be borne by X."

Do you have any concrete examples where a third party library turned out to be the major factor of a project failing? I've done plenty of projects that rely on a large number of at least moderately popular open source components for example and I've never once had a major problem. Obviously all assumptions carry risk, you should have good contract terms etc. but from experience this element is not a high risk at all. Clients tend to understand the elements that are out of your control. Changing requirements is a vastly higher risk for example.

If you are contracting, and the libraries are also custom, and built by different contractors, what you'll get out of your tests is just a tool to accurately know who to blame: I'd much rather have that be done by whoever is in charge of the entire project, but hey, if you have to, you have to.

However, when you are in charge of a project, and it's your ultimate responsibility whether it success or fails, testing libraries and services can be crucial. I once worked on a large distributed system that was built around a third party distributed database. Said database had a bunch of features past what a typical distributed database promised, and a good percentage of them were part of the design.

The people in charge of the project at the time never bothered testing those features thoroughly, and spent a lot of man-years building. When it came time to have everything working together, a whole lot of those special database features just didn't work reliably or quickly, there was no observability to be able to identify the problems, so the project was an unmitigated disaster, and the reasons for this weren't even clear. I came later to try to rescue the project, built a bunch of testing, and got a good picture of which features worked, which didn't, and evaluated the workable uses cases. Then came a redesign around what worked, with me reimplementing some of those broken features in ways that compensated around the database's practical limitations.

Property based testing and load testing can be ignored for toy projects, but you'll be happy you've done your homework early before you build something that will be used in anger.

I just wish property based testing libraries were built by people with more industry focus, instead of just those interested in thought experiments and academic proofs: The ones I've used often needed extensions to make them have industry-level ergonomics, instead of behaving like someone's Ph.D thesis.

What in your opinion is the most important missing extension at a high level?

I've thought about this same thing before, also in the context of not contractual work, but longterm.

At least 1 thing is that if something goes wrong later on, you can point to the tests to show that _your work_ is no the culprit. I'm curious to hear what the more experienced people have to say on your question.