Hacker News Reader

Comment by js8 | Hacker News Reader

> What's the gap between a Co-pilot written test and a human-written test?

IMHO the gap is that human-written test understands the (unwritten) specification (which might be as simple as "program doesn't crash"), and tests the program against it.

I am of the (pretty radical, it seems) opinion that the (especially unit) tests that just take the code as the specification, instead of having specification coming from elsewhere to test against, are useless. So you cannot build a useful test just out of the code, because you always need to test an implementation against a specification, or, two implementations against each other.

View on HN

I agree with you about ideal approach to tests and spec but sometimes the "specification coming from elsewhere" is pretty vague, incomplete or simply lost in time.

In that case, if I have code that satisfies that specification as demonstrated by say acceptance testing or years of use, I will happily opt for test that take the code as specification. I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on. Worse case scenario they should keep the program as broken as it is now or has been for years.

> I am not using those tests to prove the code "correct" but to prevent regressions during refactor or new feature development later on.

Then you don't really need to have the tests written, what you need to do is to compare the output and behavior of the two code bases before and after refactoring. I mean why use a low fidelity approximation of the original (i.e. test that only works in some cases), if you already have the original at hand? (After all, how can you predict ahead of time, which features of the original version will need to be tested as preserved in the new version? Isn't writing regression tests in advance a form of premature optimization?)

From the view point of testing, you're comparing two implementations. So again there needs to be some understanding which one is the correct specification - the original one or the new one?

It would be more useful if the industry actually treated the two cases - i.e. testing against specification and regression testing of the refactored code - as completely separate, instead of trying to push unit tests for everything. Because in the latter case, ideally you can prove that the refactored code is doing identical thing, which is stronger than testing.

Completely agree with this approach - recently I find that most of the tests I write are so-called 'integration' tests. I find these tests not only get more code coverage for fewer tests, but they more closely represent the real or implied external specification. In other words, they do a better job of verifying 'is the code correct' rather than 'is the code we wrote the code we wrote'.

I think the specification from elsewhere is all about the business value that the code delivers. At the end of the day, the business doesn't care about how exactly your code is decomposed into classes/functions/etc, the business cares about 'can I register a customer' and 'are customers prevented from writing to resources they only have read access to'. Of course there is some requirement for the code not to be a complete mess, as this has impacts delivering features and reducing bugs down the line, but with tests that sit closer to the value the code delivers than the structure of the code, you're free to refactor and the tests will tell you what you've inevitably broken. I find this considerably more valuable than unit tests.

Do you know of any good resources to read about testing? How did you learn what you know?

I only know very little. But we had a great guy in our company who was a big proponent of rigorous methods even for non-critical software and he made a very interesting training about it. Here's the list of his recommended resources:

Increasing QA maturity level:

– B.Beizer – „Black-box testing“ ($), „SW testing techniques“ ($)

– ISTQB Foundation level

– RTCA DO-178C ($) / EUROCAE ED-12C ($), DO-248C

– Joint Software Systems Safety Engineering Handbook

Test analysis specification-based techniques (EP, BVA, EP DI, STT, TCl):

– ISTQB „Advanced Test Analyst“, „Advanced Technical Test Analyst“ courses and Syllabi

– BS 7925-2 „Standard for SW Component testing”, ISO/IEC/IEEE 29119-4:2015 „Test techniques” ($)

– IEEE 829:2008 „Standard for SW testing documentation” ($)

Test analysis failure/risk-based techniques (FMEA, FTA, RCA):

– C.Wilhelmsen, L.T.Ostrom – „Risk Assessment: Tools, Techniques, and Their Applications” ($)

– C.S.Carlson – „Effective FMEAs” ($)

– MIL-STD-1629A „Procedures for performing FMEA”

– NUREG-0492 „Fault Tree Handbook”

– NASA Fault Tree Handbook with Aerospace Applications

– DOE-HDBK-1208-2012 Volume 1 „Accident and Operational Safety Analysis Techniques”

– TOR-2014-02202 „RCA Best Practices Guide” ($)

Reply on news.ycombinator.com