The logical contradictions eventually overcame me.
The logical contradictions eventually overcame me.
The point is that when you do integration tests, you will test the underlying classes the way in which it actually matters. You can't test every possible condition a unit can have, but you can test for the most likely.
IME unit tests can only effectively substitute for integration tests where you're testing logical/algorithmic code with simple function inputs/outputs.
If there were enough code involved in a test that we could meaningfully refactor it while keeping the test green, we would call it an integration test.
When testing NAPI like to use edge cases like the company with the longest name eg "Donaudampfschiffahrtsgesellschaft" or the Famous Welsh location "Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch"
I am pretty sure I crashed an A17 clearpath mainframe by doing aggressive testing
* Have at least one unit test for each nontrivial unit, but no unit tests for the trivial ones. But, in these unit tests, use mocking and stubbing only when using real objects isn't feasible, or very inconvenient - isolation is nice to have, but generally overrated in my experience
* Create as many unit tests as necessary to feel confident about the edge cases for the most critical units
* Implement higher level tests covering the main use cases for the interaction of all the units
> In a given computing context, the exact function to be called is determined at run-time and cannot be deduced from the source code [in OOP languages] as it could in FORTRAN.
This is not necessarily true, and I don't think this was true when this was written, either. (The Wayback Machine first saw this in 2014, and the paper doesn't date itself.)
Most member function calls in C++ (non-virtual ones) can be resolved at compile-time. Where idiomatic C++ could would use a virtual function would require some equally uninspectable construction in any language, because one would use it when one needed run-time switching. Rust lacks OOP in the usual sense, but if I needed a virtual function of sorts, I might reach for a trait type, which also wouldn't be inspectable (beyond whatever semantics the trait establishes).
There's Java, JS, and Python, I do admit. And I find my unit tests often fill in the static analysis I wish I had. But the truth is more nuanced than I think the author conveys.
> Unit tests are unlikely to test more than one trillionth of the functionality of any given method in a reasonable testing cycle. Get over it.
> (Trillion is not used rhetorically here, but is based on the different possible states given that the average object size is four words, and the conservative estimate that you are using 16-bit words).
This is as vapid a definition of test coverage as the line coverage the author derides in the prior paragraphs. A trivial function,
fn add_these(a: u64, b: 64) -> u64 {
a + b
}
could not be adequately unit-tested, because I could never cover the many trillions of input states it has.I think there's always an implicit line to "well, if the code under test is absolutely nuts", it's not the test's fault for not catching it, and that if the function under test (with the same contract as the function above) is errantly coded something like,
fn add_these(a: u64, b: 64) -> u64 {
if a == 11771008849893880921 && b == 5668622331333919113 {
0
} else { a + b }
}
then what amount of testing is going to catch that?> I define 100% coverage as having examined all possible combinations of all possible paths through all methods of a class, having reproduced every possible configuration of data bits accessible to those methods, at every machine language instruction along the paths of execution.
While this somewhat contradicts the earlier example of states. I don't think we should aspire to this, since I think many "combinations" are not interesting to test; they're just pieces of the code that really don't interact; testing them is redundant. Yes, this hurts our ability to formally define a protocol for testing or not testing, but it prevents the inevitable combinatorical explosion.
> Business people, rather than programmers, should design most functional tests.
Absolutely not. A large part of my job is coming up with the actual requirements of the system from the vague machinations of business people who barely know how a computer works, let alone can specify salient requirements. How are they to write the tests, when they can't clearly articulate the requirements?¹
> Turn unit tests into assertions.
But I want to ensure these don't fire under normal circumstances. We do this, in my current code base, and we get notified of them, and of any crashes in general. But it's a failure that was noticed, and might have downstream consequences, as opposed to one caught by a unit-test.
Then the parts about Eastern Europeans with bad Internet make better programmers because they had to think instead of using the Internet, and how he grew up under similar circumstances. Real Programmers!
¹I'm not saying business types "should" (or should not) be able to articulate requirements/specifications.
Additionally, some fuzzing tools can actually to branch analysis and target your "rare" codepaths. Now, sure, figuring out how to hit some codepaths basically requires brute force - it's not going to magically figure out an efficient way to collide SHA1s if that's what it takes for your test to fail - and you need a way to differentiate good from bad behavior (this might be a reference implementation, or this might be as simple as "does it crash?") - but you do have to get crazier than add_these before you hit the limits of even existing tooling to catch bugs.
Mutation Testing could. :) https://ai.google/research/pubs/pub46584
At least, I didn't take away more than 'throw away tests that haven't failed in a year' and I don't agree.