One third of C++ devs in this survey don't write unit tests at all? That's insane in this day and age.
Integration tests and fuzz testing is where it's at. Anyone can add or improve an integration test within an hour and find several bugs that real users would encounter.
Well, that might be the problem. Testing "for bugs" isn't a terribly helpful activity. Testing for behavior is much more helpful, and unit tests can definitely be applied to that aspect of the system. Of course, it requires you to properly decompose the system into subsystems (modules) with loose coupling that can be individually (or in small sets) loaded into test harnesses for testing.
I have written tens of thousands of unit tests in my career, and by far the most useful ones are the ones that point you to the exact subsystem in which a bug you've just introduced during refactoring/re-implementing lies. Bad unit tests are brittle "change detectors" and make you constantly adjust them for no real reason. I find mocks and interaction-based tests (rather than simple input/output tests) are this way. I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests.
If your unit tests are garbage, it's because they were not thought out very much and commit sins.
This is why I prefer integration tests. Most of my projects interact heavily with a database. Making it an integration test unless I mock it out. Which I don't do, because in my experience, doing so tends to hide lots of bugs.
IMO integration tests are great for making sure that your system actually functions with all its parts together. Nothing but an integration test can do that. But unit tests serve a different role: reducing the scope of the system under test so that you can pinpoint exactly what part failed when it does fail. They help you iterate quickly in refactoring, and when something goes wrong in production (or a bug report from the field), they narrow the cone of uncertainty by eliminating from consideration the parts of the system that are working correctly. In my experience, this rapidly cuts down on the amount of wild goose chases one goes on and overall makes a team more efficient. Then once a bug in the wild is found, the system responsible for the failure gets new regression tests and unit tests. Sometimes, I wrote unit tests for units I am suspicious of as I am debugging, and even if those tests don't find a bug, I might keep them.
I worked for 6+ years on V8 and the parts of the system that were the least unit tested seemed to be home to far more production issues than others.
It also helps a lot to keep your unit tests running super fast. Under 1 second is ideal.
As I say, some people will still think of this as integration testing just because there’s a lot of moving parts working together. But often you’re just testing fairly pure code with no external or IO dependencies so I’m happy thinking of these as unit tests still. Where those types of dependencies exist I do still mock though.
It's fascinating, I agree with the rest of your comment wholeheartedly but am a heavy mock user. Much of the code I interact with has complex logic with well-defined and discrete interacting objects. Without mocks, the combinatorial explosion of input/output behavior means tests are extremely difficult to write, or more likely, are written incompletely, without properly enumerating the invariants expected of the interfaces under test.
I'm currently working on a complex system right that's safety-critical but also SOTA, and full of domain experts who are decisively not engineering experts (among other things, engineering expertise is part of what I bring to the table). Well-defined invariants and narrow unit tests that enforce them are a _god-send_, and I'm literally not sure we could function without them: as we're constantly refactoring the system to work in different ways, automated testing of invariants enables a whole universe of experimental changes that would otherwise carry a great risk of introducing bugs into other parts of the system. Our system is far too complex for integration tests to cover a fraction of the completeness that unit testing enables (though we also have simulation/integration tests, of course).
Of course, I am sure many of those survey results are people that don’t write unit tests when they could and should.
I've seen 100%+ code coverage on cross-compiled bare-metal C++ embedded code. If that can be tested, then anything can be tested.
There are broad classes of software, notably high-performance data infrastructure, where unit testing manifestly doesn’t make sense. So people don’t write many unit tests even though this is mission-critical high-reliability software. Other kinds of testing is extremely thorough though.
The kinds of code units that are not unit testable are those where the correctness of the implementation behavior is almost completely contextualized by the behavior of external code units. As in, the unit test may be correct or broken based on the behavior of external code modules, not based on the inputs to the module. Software that is very schedule driven, like modern database kernels, has this property generally. Good integration testing picks this up very well. Writing a proper “unit test” under these constraints is effectively equivalent to a relatively comprehensive integration test.
And for parts of the code that can be unit tested, I tend to use exhaustive or quasi-exhaustive testing of their properties as part of release qualification, which is too expensive to be a true unit test but picks up more obscure errors than unit testing is ever likely to find and serves a similar function.
I believe in brutal and thorough testing. But unit tests specifically are mostly a waste of time for some of the common applications where C++ excels as a programming language.
Surprisingly 93% of Java users surveyed write unit tests though. I thought Kotlin would be similar, but for some reason JetBrains doesn't show any responses for Kotlin users.
Java percentage is unit test is so high most probably because it's an enterprise language. And the whole unit test craze started with Java and jUnit.
I'm not sure that's the main factor. Compared to C# & C++? I don't think "enterprise" has much to do with it.
Case in point: I'm a hobby C++ hacker, and I read this. I'm a long time pro Python hacker and would not have opened something similar for Python! I'm like "the last thing I need to read about in my leisure time is what other people are doing with Python, that's WORK." haha.