Thoughts on Testing
brandons.me
brandons.me
However tests are often used or at least attempted to be used as a crutch for poor language/tooling/architecture decisions.
The path to code that breaks less is in good decision making (ideally early in the process) to ensure as much as possible the code you write is "obviously correct". Part of this is languages that protect you from silly mistakes, i.e statically compiled, strongly typed, etc and tools/libraries that help you write good code and make use of constructs that are easier to make "obviously correct". i.e sequential/blocking code styles vs async/callback styles. (obviously there is no such thing as "obviously correct" in an absolute sense, hence the air-quotes but some approximation thereof)
Similarly architecture plays a big role. If you concoct a pile of separate micro-services that then all need to be integrated tested together because you didn't settle on an IDL and contract between them etc then you are in for a world of hurt.
Some of it just comes down to principals also. Prefer idempotency. Try design around streaming systems instead of job queues if you can so you get replayability out of the box. Focus on data first, a good schema and using your DB correctly can save you thousands of lines of codes and all the tests that go with them by preventing bad states from ever possibly existing, etc.
If you don't give yourself a good foundation and try to "fix" it with tests and "process" you are doomed to a bad codebase that will just get worse with a test suite that seems to grow forever that either doesn't meaningfully reduce defect rate or if it does then it drastically reduces velocity.
The golden happy path has tests and process but only just enough of each to keep things in check, the good decisions are meant to carry most of the weight.
I agree with the overall sentiment but I don't think this is a good example. If `await foo(); await bar(); await baz();` is subtly wrong, then `foo(); bar(); baz();` isn't obviously right. Continuation passing style is a minefield, of course, but callback hell is a historical artifact, not an intrinsic feature of asynchronous code.
If you `await` every async call then sure, at that point you have made the code sequential within that logical thread of execution, if anything proving my point that it makes it easier to be sure it's correct.
I would say that once you aren't doing that, i.e doing real concurrency within a single "thread" of execution then that is going to be less obviously correct then simply not being concurrent.
That said, broadly speaking async/await is sequential/blocking enough for my taste.
There are runtime/language specific problems with it that reduce it's obvious correctness IMO (unhandled promise rejection when not immediately awaiting on Node for example) but it's not an API problem.
TLDR: async/await good.
If you can unit test something as a unit then do it... the moment you put a MOCK in that code is waste.
That means to test your system you need to go end to end. API -> processing -> storage -> response.... Write the records and read them back. That also means your UI needs to do it to! JS keeps adding features in the browser, and none of them result in reliable testing. We need to do better.
But end to end testing is hard, our environment is complicated, it takes too long. Fix it (its fixable)! Bound your test system to run from nothing to complete in under 20 mins. Treat failed tests and long runs like production outages.
Write your end to end testing as stand alone. That means if you move your code base to a new language, your tests should still be valid and running.
Does this mean dont write unit tests? NO you should have some unit tests. Things that matrix out well (think email validation), are the softballs that should have some unit tests. Highly contentious code that business "logic" that is a dense ugly switch... unit test that, build the mocks, because if you have to touch that you want every confidence that it will pass your E2E tests.
Watch where you duplicate: Its the job of the front end to fully exercise its code. Its up to API devs to fully test their API's, to hit all their boundary conditions. Will there be duplication, sure, but its going to make clear where breakages are happening between layers.
This can be necessary but very tricky to achieve without mocks/fakes or whatever it's called.
You don't.
1) The outcome of the tests will be: 0.1% There is an issue. 99.9% Maybe there's an issue.
2) Finding concurrency issues needs fine-grained targeting. You can't just write a "does anything race?" test. Testing won't reveal the location of the races. You already need to know about the race condition to be able to write a test for it. And once you know it's there, what are you testing? Are you making sure that the system does the wrong thing?
Say there is some weird behaviour and you suspect that it's due to a race condition. Maybe someone forgot to do "select for update" and you want to write a test to confirm. How to do that?
Even if you know 100% that it's a race condition, you still might want to add a test to prevent it from happening again no?
That you've fixed it.
I disagree. I mock the interfaces to other components that have clearly defined output in a clean architecture to properly unit test (i.e. a use case). Everything else causes components to be mixed and things to be tested again and again, causing complex and glued together tests.
It all depends on proper architecture, which is hard to get right though. We're using kotlin (sealed classes ftw), hexagonal arch + ddd (use cases) and graphql/rest on top. Everything split up by gradle multi modules with architecture tests to verify that modules have not been mixed. We also have a domain module with plain kotlin that implements business logic free of implementation details. With this high separation of concerns unit testing is the only thing you need to do (automated).
Some parts of the algorithm were easy to test, others very hard. On the easy part was loading and saving the file. I tried to test as best as I could, but the lossy part was not really tested.
Then time to benchmark the algo on some images. The results were kind of OK, but some images compressed much better than others at low bit-rate, while the content did not look much different.
On square images, everything was fine. On rectangular ones, at high-bit rates it was somewhat fine, but strange patterns like ghosts of another part on the image started to appear for lower and lower bit rates.
Of course, I searched on the mostly algorithmic parts what was going on, and failed. Only after a holiday break and fresh eyes did I found the root cause: inversion of height and width in some part of the loading AND saving.
I had "end to end" test of the loading and saving, but the error was symmetric, so undetected. Young engineer error...
Changes in interfaces or behaviour of a module won't break the unit tests of modules that depend on it. So you must rely on either typechecking (not available in all languages) or integration tests (which will then trigger a change that will cause the unit-test-with-mocks to break).
If there is neither to help, you code just breaks in production.
Not only that, but this can cause some design ossification: classes that have too many dependents become very difficult to change, since you now also have to modify all mocks of it.
[1]: https://docs.python.org/3/library/unittest.mock.html#auto-sp...
As with most things in life, I find the best approach is seldom at an extreme. The time spent on writing tests should only be spent if those tests help you achieve the overall goal better than spending that time writing application/library code. And vice versa.
With that in mind I feel I get the most bang for my buck primarily writing higher-level tests like integration tests. These will test that the core functionality works, and cover larger parts of the code-base.
Like Lego, the pieces might be perfectly in spec and the error might be in how they're put together. Thus unit testing will only get you so far, and I mostly reserve unit testing for smaller pieces of library-style code, especially if there's tricky edge cases involved.
I would also add that I prefer higher-level integration tests because I want the tested code to be malleable in case I need to refactor in the future. To me unit tests take away some of this malleability.
I write unit tests for the same cases you do, however I also tend to write mostly unit tests for regression testing, which sorta breaks my own rule.
Testing internal details is technical debt.
If your change has not effect on the public API contract... why are you doing it? Think harder...
and previous discussions on HN: https://news.ycombinator.com/item?id=15591190
Related, but writing a regression test for every little bug you fix isn't always worth the time either, like when the test is complex to write, the bug isn't likely to happen again, and the consequences of the bug aren't costly. It's rare to write a test to confirm a typo in some copy/text was fixed for example.
Sounds obvious, but I've been on teams where you're seen as a cowboy if you even try to discuss the downsides/costs of testing.
(put it another way: if the programmer introduced a bug by mistake, probably by writing a test it will test more than wanted by mistake...)
I'm aware of this, it still comes down to weighing up the costs vs benefits. It's not practical to write a test for every small behaviour of a complex app.
After all, if a customer sees the same widget break the same way 3 releases in a row, the last thing they want to hear is: "Well, actually, that just looks like the same bug, but really..."
I mentioned to weigh up how likely the bug will happen again and the consequences of that specific bug. A typo isn't likely to impact trust nearly as much as a crash.
In what scenarios do you not write tests? Have you been in a situation where writing the test would be a lot more work than it's worth?
If the functionality is changed, the tests will have to be changed to reflect that. One feature of tests is that they are are documentation for the expected functionality of code, so naturally it needs to be updated when functionality changes. It is a feature in my opinion.
I have been writing YAML based tests which rewrite themselves based upon program output in these cases.
Change the code -> run one script and now the test is updated. There is near zero test maintenance cost for surface level changes like this message or a tweaked JSON output from an API.
I call it snapshot test driven development.
I agree with the overall sentiment - testing ought to considered an investment that provides a recognized ROI.
I think snapshots definitely lower (but don't eliminate) the cost of a certain class of tests, and can sometimes tip the scales from "not worth it" to "worth it"
There is a hidden cost to snapshot testing - theyre highly susceptible to flakiness. The tests have to be hermetic and the all the code has to be deterministic or it doesnt work.
Achieving those two things is not always easy.
IMO, if you've decided to test flaky behaviors of the code anyway, you may as well do it in a way where updating is really easy
int f(int n)
{
int result;
assert(n >= 0); /*precondition*/
...
assert(result >= 0); /*postcondition*/
return result;
}So, responsibility for determining the correctness of the return value and what to do next (return error, retry, ..etc). falls to the caller.
Just wanted to throw that out there.
I write mostly small pure functions. In those with proper preconditions it is impossible to have invalid output since the output for a given set of inputs is always the same. So, if the inputs are valid then the output must be valid as well.
Where I can’t write pure functions the post conditions are more along the lines of evaluating status codes and/or response bodies. Those I have found make more sense to allow the caller to evaluate and choose what to do in the context of the calling action. It isn’t that I am duplicating post conditions but that it is really better handled with more context as to what is being done since different callers may have different definitions of a correct response.
It is entirely possible though that in different domains or languages that post conditions make more sense. I am not saying their bad by any means. It just in what I work in currently I tend to just turn those post conditions into preconditions for the next action or gating some recovery logic.
But, I have tended to prefer flat call structures and keep retries and error handling logic at the application/service entry points if possible.
Programmers mis-using the word "test" is one of the great forces against good testing. If you have any intention to test your products, programmers, then you must investigate your products. This does not mean setting up an "optimal" number of automated output checks any more than doing police work begins and ends with setting up CCTV cameras.
Test is a verb, primarily, not a noun.
Younger me hated tests. More experienced me sees tests as a necessary part of building high-quality software. Younger me didn't like tests because:
* Writing the test can be more difficult and time consuming than writing the feature you are testing.
* Tests redefine what "done" means for a new feature by adding "tests pass". That means it will take more time and attention to detail to finish.
* Finally, tests introduce a new constraint: you have to take tests into account when you change code and that slows down development, too.
All of the above said, there's definitely a sliding scale of too little/too much when it comes to tests that ranges on complexity and the developers working on the software.
Good programmers understand that tests allow them to move on safely to the next thing, as professional if you are not doing this you are creating technical debt.
It feels like your analysis is based on having to add tests without enough support from a skilled tester who is empowered to drive testing, if the tester is just told to rubber stamp code without delaying deployment they wont be able to do much. Thats not really the fault of the tester that is a choice to deprioritise test.
2) Understand what is on the other side of that boundary (human? machine?) and attempt to replicate as far as it practicable.
3) (optional, experts only) identify internal boundaries that may benefit from testing in this manner. Beware that artificially erecting such boundaries in inappropriate places may seriously compromise system development.
I write tests out as flat requirements text.. No code, just documentation. Quick and dirty.
"When the user puts in a bad password, it returns an error."
Then convert this to real tests, and tada, I have red tests.
The code almost writes itself.
- Test give the ability of other developers to be productive on the project faster. Having tests tells other developers the intended behaviour and notifies them when they have broken it.
- Flow on effect of slow or large test suites. It demotivates developers to write new tests, run existing tests and accept failures if there are too many. This then leads to a lack of confidence, trust deteriorates between team members and development pushes towards going faster than being stable. Eventually, once enough bugs occur or a large incident happens, then you go back to looking at tests again.
If you have fast and reliable test suites, a developer wants to run and add to them. Developers feel that this is a high quality project that needs to be well maintained. This culture then permeates into other areas of your business.
(Sorry for the self plug) This is why I created Data Caterer (https://github.com/data-catering/data-caterer) to try provide a fast and reliable tool to help with end to end testing.
Trust us, tests exist and they have a plan
Also the article isn't talking about "testing functionalities", it briefly talks about code that is redundant to test in certain ways (eg: unit testing), not "functionalities". There's way more nuance to that.
An example: you could even have 100% coverage via integration or end-to-end tests, but still have zero unit tests (EDIT: or the inverse, you could have zero integration tests). The discussion is not just about the what but also about the how.
it is more likely to easily write a unit test for a piece of code that is well architected and written using TDD and also to safely modify it, than to modify or write test for a 50 lines of code beast self-contained, undecoupled method
Writing unit tests should not take time, should be part of the time needed to write the method, if it's hard to write unit tests for a unit of code, then it's most likely that the code needs refactoring
So let me ask then, if there's 0% unit tests, how do you ensure maintainability, cleanliness, decoupling and cohesion of code? By indenting the assignment columns? :D
Please don't interpret examples of extreme hypothetical scenarios as suggestions or recommendations.
EDIT: I guess the broader point of my comments is that you shouldn't assume that others are taking extreme positions. Again, as an example, it is also possible to have 100% coverage with only unit tests! But that's just an example of what's possible, not a recommendation or prescription. And I wouldn't recommend extremism in either direction. As always, "it depends".
EDIT: I can only think of reasons beyond technical, in case of limited time to market or tight deadlines, but in that case it's tech debt tracked by tickets and docs, still going to be subject to test-writing
Are there really (mainstream, well-regarded) books which promote the idea of endlessly ideating and writing tests? This isn't a narrative I'm aware of?
Having a comprehensive test suite doesn’t mean the code is infallible and you won’t have to fix bugs, but with every bug a new test increases the future resilience of the code.
And finally, tests enable new contributors to work confidently from day one.
I agree that tests are a liability, but the amount of testing we do in software is still laughable in comparison to any other engineering discipline.
In practice, we tend to have a pile of tests that helped the developer write the code, but now are simply redundant. Then you have all the tests that assert on internal details (strings in error messages anyone?).
My favorite guide to testing well is in “Large-Scale C++ Software Design” by John Lakos. The problem at its heart is one of engineering practices. Structuring code bases physically into definable components/modules, interacting only via defined api’s/interfaces and providing test suits that both document and verify the contracts. Done right, refactoring/rewriting involves no, or extremely minimal, changes to the tests.
It can be done right, but it requires significant discipline.
1) Give you confidence that if you refactor something, when you do something silly it'll break the tests and show you where you went wrong quickly
2) Allow you as a developer to write new things in isolation, so you can run the code without needing to jump through any hoops to see that it's correct
I tend to write tests when the thing I'm developing isn't simple/cheap to access. Writing tests should help you do things faster, as well as stopping future you from doing daft things. Writing tests dogmatically however, is a waste, and if tests are hard or onerous to write, it shows that the application is probably poorly designed and architected.
Exactly this.
If I run into an issue when using OSS, I tend to try and look at contributing a fix back.
The projects where this is most successful are those with a good range of tests - I don't have the time to sit and learn the workings of a project inside out for the sake of a single fix, a good test suite helps reassure me (and them) that I've not inadvertently broken something.
That same benefit exists for new starters working on closed source codebases - they can hit the ground running much faster, confident that tests will help make sure they don't accidentally blow things up.
But, the OP is also right that tests need to be written in the correct way - built based upon the intent, rather than the code that was actually written (where I can, I tend to try and write a test first - even if I might later need to go back and tweak it)
You cannot provide easy and straight answer "how to write a good test", just as you cannot provide easy simple answer to "how to create good code".
It requires knowledge, practice and experience. It also requires case-by-case study. Not every problem is a nail.
- having said that I have written some pure unit tests, but more often I wrote functional tests, even for "units"
- shifting towards integration tests, may make your tests grow bigger in scope, in initialization, in result checking. There are always trade-offs
- to have seat belts in a car you also need to design them, and produce them. It is really beneficial to have them though, and you should not take into account scenarios without them
- more often I tests API
- they produce sense of security, but they have never ever produced certainty. No one said to me "everything works because we have tests", therefore this argument seems to by more hypothetical then real
- without tests I would have troubles with writing stable software, and I think I am not the only one
Then I suppose we can agree
- writing some tests is a required
- writing tests for critical code parts is required
- updating them is required
I am also someone who, at interview, is a proponent of TDD - but in reality, and after landing the job - has never used it nor worked anywhere that has used it where they say they do.
With that in mind. How the fuck do you manage change requests with TDD.
Riddle me this… given:
We change the code behavior, and now our test fails! But the failure isn't because we introduced a bug, it's because we changed our mind. The new message is now the correct behavior of greet(), and so the test is now incorrect and needs to be updated. Time spent "fixing" the test is pure overhead.
Do a whole back and forth with whomever comes up with the specs?
Or
Edit my code so on the second invocation it presents the correct message, assuming tests are run in a consistent and deterministic manner this would work.
In this instance it would be pretty clear there is a wrong specification so I could go back and ask someone for clarity.
But what if it is not clear. What if it is some intricate or subtle behaviour change where you can’t use your intuition to figure out what is a correct test or not.
How do you even know if the tests that you have written correctly describe the desired behaviour? Pass them by the decision makers who are most likely non technical?
The only thing that important is how your end users interact with your services and what is a dealbreaker to them. Everything else is a nice to have. These can not be tested in any way other than having your application in their hands. This mo has served me very well in my career so far.
Sometimes almost or even completely useless tests are written. Its a cost I can live with.
While writing tests alongside writing production code you might become aware of code that does not conform to your own qualitative measurements. SRP, DRY etc. That process, writing tests while you write porduction code, imo, has a tendency to better your design. But I never look at design as being done, I look at it as being good enough for the time being.
If you write test after you write you write your production code, bettering your design becomes more difficult, since the code is already written and you might need to make larger changes that will adhere to how you really wanted the code to look like.
Tests should give you confidence. Without them you are clueless and have no confidence when you start making changes. You cannot juggle with, on a cognitive level, that if you apply a change to your code, that whings will just work. Tests gives you that confidence because it gives you fast feedback.
You can not argue that removing dependencies and therefor mocking is a possibility. Unless of course you have no dependencies, like the calculatur or greetings funtion in the post.
You can move your dependencies around and perhaps invert them, but you cannot remove them. A good example is that instead of injecting external dependencies to your own classes, inject objects which represent a dependency in isolation.
Then you can control the isolated object which is a representation of an external dependency but do it without mocking.
The best book I have read about testing is Vladimir Khorikov's https://www.amazon.com/gp/product/1617296279/ref=as_li_tl?ie...
I think the best defense of unit testing I could make in response to the article, is admitting that TDD makes it painful to write tightly coupled code, and that's the point.
If you want to write tightly coupled code (maybe a quick demo of a feature?), or keep the tightly coupled code you already have, don't test it!
The author gets into this a bit, but I'd say the reason they correctly identify testing a UI as painful, is that a lot of UI code is actually data, not behavior. One could argue that enforcing a clear distinction between the two is a benefit of TDD.
That's also the problem with TDD. Another word for "tightly coupled code" is "cohesive code". There's a tradeoff between coupling and cohesion. The problem with TDD is that it can make people decouple cohesive code just to make it testable. I have seen this many times where the only use of some abstraction/indirection was the test. These abstractions/decouplings then mount up to make the codebase much more difficult to understand and maintain than a simple cohesive codebase (with decouplings only where necessary).
It really depends on the test tools you have. If you have good mocking tools, and things like test factories, then it becomes quite easy.
* When I add-to-cart, it should update the cart total
* When I add-to-cart, it should not decrease the available stock
* When I purchase, it should decrease the available stock
While these might be tests, they are also a form of live-code documentation about what the author intends. If you find later code that violates these expectations, you know that the old behaviors were intentional and you are updating expected behaviors that were not a side effect.
I've had such colleague during my career, who declared "who to write tests if I can write new features?". That will be a valid question if he could write code without tons of bugs. When he left the company velocity of the team increased.
That's really not what technical debt means, but yeah, you have to maintain tests. There's a cost.
> So what is the optimal number of tests?
NaN. There is no right number of tests. And more test cases does not mean "more coverage."
> while tests cover individual inputs
Depends on the kind of tests you write. When I'm doing property-based tests I like to randomly sample inside equivalence classes to test that I'm setting boundaries correctly. Picked up the habit when I saw that catching bugs when we moved from 32 bit to 64 bit code.
> The fibonacci sequence is well-defined, it has a clear definition, it will never change
Then you don't need automated tests, do you? Manually show that it works once then have automation that diffs that code to make sure the promised "never changing" really happens! Not what I would do, but different strokes for different folks . . .
And I wouldn't do it because the implementation of something non-trivial and not made up as a straw man might have a valid implementation change. Which is a great reason to write behavioral tests for the function.
> calculation()
The proposed tests don't test the interesting stuff. What type can n be? Can the function overflow? Does it slow down when one type is cast to another?
Also, why do you have a function that just returns n * 2 + 6? That's gotta upset those anti-clean-code folks you work with! Ugly high-maintenance tests can point to dev code smells.
> isEligible()
This function does something so trivial that if it were actually used, I bet it's not used everywhere for the check. This is the kind of thing where people use isEligible() in places and just hard-code checks against the age in others. And whenever the age changes, something will break.
Again, the tests seem dumb, but I bet there's a better way to write the production code.
> Is the test simpler than the implementation?
Seriously? If so, that might be a sign that the implementation is too complicated. Functional tests will typically have some kind of setup (possibly complicated) then the simple thing they're checking, in your ideally easy-to-understand function.
> Mocking adds a ton of complexity and reduces the amount of functionality actually being tested. It makes code both harder to test and less valuable to test.
Without knowing more about how you've done mocking and why you say that, I can only sound like one of those zealot snobs when I look down my nose and say "you're doing it wrong." I don't want to do that. So without a solid example here, I can't really respond.
FWIW, I mock to see how the results of different wire calls will affect the code. A small, well-defined set of results. I'd rather do this in unit testing with mocks than stand up the service and surround it with fakes because that's a lot slower and more brittle and I have to think more. And I'd rather stand it up surrounded with fakes than test in a real, full environment because that is slower still, even more brittle, and requires some kind of magical powers to debug.
> Integration tests > almost by definition you aren't really mocking systems, you're testing the interactions between them that are otherwise hard to test
I bet you have never heard of contract tests. They're a faster, lighter, less brittle, less brain-hurty way of testing interactions between services. I swear I don't get a commission from Pact.io (a SmartBear acquisition), but you should check them out.
> UI tests > Often change output/behavior intentionally (leading to flaky tests)
That's not what a flaky test is. If you change some code's behavior and don't change the tests and the tests break, they're good tests - fix them!
When a test passes . . . except sometimes it doesn't, so everyone says "run it again," that is flaky. And flakiness sucks. Often caused by not waiting for (or being triggered by, depending whether you're doing polling or push/promises/etc.) the right condition before taking the next action.
> have trivial logic that's easy to verify by looking at it
So, again, I assume you have tests that diff that section of code to verify the tacit promise that the code won't change.
> I don't hate tests (I promise, Avid!), I just try to question them on a case by case basis, and I haven't met a lot of people who do.
I'd suggest trying to understand the tests instead of questioning to prove they're not useful.
> They have a very real organizational cost
Yes. And so does bug fixing. The biggest cost of bug fixing, IMO, is finding the places where there were contradictory expectations. If only someone had coded those expectations as tests . . .
> makes that coverage number go up, so it feels good to add them.
That's a process smell. And I don't just mean the gamification. Code coverage is really important but not for rising numbers. What matters is what's not covered - there be bugs!