Stop being frantic about code testing | Hacker News Reader

Stop being frantic about code testing

unixsheikh.com

20 pointstodsacerdoti21 comments

I'll give this one an upvote just for the discussion of an article written by someone that has a serious bug up their butt about testing (if you will allow the pun). As another commenter points out, it's Poe's law in action, though I'm leaning toward "they're serious, aren't they?"

My only rebuttal is that if things can be tested manually, as is the preference of the author, then you're either:

1. Building a toy app, or at least one that doesn't do more than a handful of things

2. Are not testing as thoroughly as one might think

3. Are going to soon be overwhelmed by the testing workload

4. Are going to start missing bugs, especially regressions

5. More than likely all of the above, with the possible exception of #1

Except that most major software you use has only limited automated tests that are nowhere near comprehensive.

For example large parts of the Linux kernel are only covered by manual testing and some automated bug finding tools, not traditional unit or even integrations tests.

The answer in practice for most software is that the testing simply is not thorough, bugs are missed, and regressions slip in (and get fixed once noticed).

And that is all not necessarily a major problem. Of course, having some level of automated testing, even if somewhat cursory is highly advisable. At the very least least for any smaller components that are easy to test. But a comprehensive test suite for each component is not at all a hard requirement of software development.

that will certainly depend on your definition of "major problem". Most software is certainly full of plenty of bugs, maybe it's still worth having it regardless, but I wouldn't call it not a problem, see also: https://danluu.com/everything-is-broken/

I think one of the major advantages of writing testable code is that you get better code and architecture for free. You end up with less stateful code, more pure functions, more clear dependencies, fewer hard coded things, flows that are understandable for humans and so on.

If a solution is so complex that writing a complete test for it is hard then the solution is probably not very good and the next coder will not enjoy debugging it.

For UX for instance I agree that automated tests are not that valuable.

> I think one of the major advantages of writing testable code is that you get better code and architecture for free.

This.

Untestable code tends to be so tightly coupled that's practically impossible to isolate subcomponents or even pure function bits. Being untestable is synonymous to being unmaintainable. Complaining about the hassle of creating unit tests is a red flag about the overall state of the whole project.

I know we're supposed to assume good faith here, but this article really reads like an instance of Poe's law.

In any case, I'm completely baffled at the suggestion to verify behaviour for each minor change by just running through every feature manually and inspecting the responses at breakpoints. Wow, I wonder if this could be automated...

And I guess "accept that failures just happen and deal with the consequences" (which is arguing against a strawman; nobody thinks we can prevent all failures, most just think we can prevent some of them) just assumes that you're not going to lose money when your checkout page is broken and it's the middle of the night, or important clients lose data.

> During development every functionality is tested each time a minor change is made.

Isn’t that much more time consuming than writing a test of any sort? Anyway that’s why e2e/integration testing with cypress has been becoming one of my favorite kinds of testing. It gives me the security of knowing that a user flow is not broken with a new change. What’s happening under the hood is of little concern and the complex/critical parts also get their unit tests.

> Isn’t that much more time consuming than writing a test of any sort?

I would go a step further and state that it makes absolutely no sense to manually run test batches repeatedly and whenever anyone touches a part of the code. If a task is expected to be automated then automating tests is the only rational decision.

Yes, it is. It also inevitably breaks down. I worked for a long time on a legacy component deep in the guts of Windows. While some new pieces were built in such a way that they could be unit tested, the core framework was a massive piece tightly coupled to itself and several other pieces such that it resisted all attempts to extract interfaces, refactor it into testable chunks, and all the other advice from "Working Effectively With Legacy Code" that I wish I read at the start of my career rather than 7 years into it. The original work was old enough that almost no one was doing unit testing at the time, and as things grew and expanded and depended on it, well......

Anyway, this was the workflow. You made a change, you put it on a machine (or, later, a VM), you ran a test pass. It was a LOT slower than running unit tests, and a lot more error prone because it required the developer to not only know what kinds of tests had to be run, but to actually run them all and not make any mistakes. There were absolutely times where:

* A developer new to the area didn't know they had to test something, so they didn't....and their change broke it

* A developer made a change and was confident their change couldn't affect an area, so they didn't test it....and their change broke it

* A developer thought they were testing something, but had the wrong bits on the machine or missed a step.....and their change broke it

* A change was urgently needed and the delta was small, so a developer skipped the testing phase.....and their change broke something

Manual testing is painful and not something I ever want to go back to. By the time I left, the general culture around testing at Microsoft was improving, but culture alone can't fix 30-year-old legacy code that was never designed to be tested, so I'm sure there are problem areas like this across the company and industry still.

My current team has a much stronger culture around testing - plenty of unit tests, some integration and prod tests, the standard mix - and it's a night and day difference. Even without doing full TDD, the fact that I can write a test case, run it in a few seconds and see it fail, change a few lines of code, run tests in a few seconds and see the test pass makes me orders of magnitude more productive. The fact that I know that the test suite will catch issues prior to checkin (and more tests will run prior to a prod rollout to confirm) makes it much easier, safer, and faster for the entire team to move rapidly and means that I can trust developers who haven't been working in this area to make changes - even if something gets missed in code review, the tests will keep us all honest.

I feel you. When working on larger applications I regularly ran into this problem:

- Adjust code - Adjust unit test - send it to CI - Integration test failed

And this was actually a good thing. Despite me being confident about my changes to the code and unit test I realized that another part of the app was failing subsequently.

Testing this manually, at least for me, leads to repetitive least-effort testing. Fill the form with whatever garbage it needs to validate and be done. It’s the shortest way to create software that sometimes works but mostly doesn’t. And those people who tested the same workflow upwards of 100 times by hand tend to lose their sanity at some point.

He goes out of his way to avoid being anti-unit testing on here, but the whole thing does come off as looking as if it's anti-unit testing. I suspect there are just as many anti-unit test posts as there are pro-unit tests, and it really bothers me that these are so common. I can only assume that anti-unit test types, like anti-OO types, were burned by a wacky unit test implementation (and there are plenty!) once and blamed it on the concept of unit tests.

In my experience, as long as each non-trivial function can be invoked on its own through a unit test (that is, without actually spinning up a live database or having remote services available), you're "good enough" on unit tests, and you can manually test the edge cases from there.

Especially since a lot of the arguments are very personal rather than general. He claims that testing will degrade the sort of flow state that devs want to achieve. But that sounds like something that just affects him personally. Testing can help that state for some people.

His way of "manual testing" is just development with breakpoints. I do not think that this person has ever worked on a software for more then a few months.

Unit tests actually let me develop a lot of stuff faster, since i start thinking about the API first, before implementing it and therefore i shift my perspective to the API consumers point of view.

Sure testing "getters and setters" is not all that helpful but i wholeheartedly disagree with the authors confidence in correctness and to just refactor the parts that feel wonky.

I try to write important software not toys.

Note that buy toys I mean something with less than 10000 lines of code. I find at about that point (no matter what language) my code changes from quick/easy to manually test and reason about, to too complex to keep everything in my head at once and so I need automated tests to keep track of the rest.

That I'm writing software that controls machines (so there are safety concerns around failure) that do important things (the wrong bug could starve millions to death) means I'm a bit more paranoid that I'd expect a game developer to be. However I'd expect even a game developer to need to do some automated testing for peace of mind.

I can relate in my distaste for the paranoia and for test code complexity and volume exceeding that of implementation and in the false sense of security offerred by 100% unit test coverage.

But manual testing is for the birds and you cant put breakpoints in a distributed cloud architecure. Better to strike a balance with a variety of testing scopes depending on the complexity of a given area of the system.

I've been considering writing the exact opposite of this blog post. Working with an application that has maybe 70% code coverage but missing testing around some major functionality like reports. We've hit the point where the rate at which we add new features has slowed to a halt as we manually have to test a bunch of stuff combined with a growing fear every time we push new functionality. Sadly we're so overcommitted on new features that we aren't devoting the time to going back and filling in more tests, but we're trying to slowly improve the situation rather than let it get worse.

Then to make it extra fun we inherited two native mobile companion apps built by contractors with 0 testing that nobody even wants to touch.

So yeah, I'm 100% joining the frantic about code testing camp. My immediate state of flow getting broken up is a small price to pay for being able to sleep on nights when we push new functionality to production.

Based on comments, I wont even read this.

There is no such thing as manual testing, that is ridiculous.

No automatic tests ==> you should go farming or something, not do programming stuff.

How do you guarantee quality otherwise? How do you know your manual tester took its insulin hit? Humans are bad in repetition.

Ridiculous.

Then perhaps Michael Bolton's rap will convince you otherwise that manual testing is an important and necessary aspect of any serious software development process:

https://youtu.be/KePxuKpwqoI

Those are not testing feature correctness, but testing feelings.

Manual "tests" are about getting the feeling on how it works, how approachable it is etc., not about feature correctness as it is described in the funspec.

Everything he describes is testing.

But it's somewhat cross-purposes talking.

When people typically talk about automating the testing, they're talking about taking rote-based manual checks and automating them. Absolutely fine, no complaints from me.

But that's not automating "the" testing. It might be automated their testing, and those two things are different.

You can easily tell by whether they're doing it to get rid of the testers (cost-saving), or freeing up manual testers to do what they're good at: context-driven exploration of the problem space.

Frontend tests (even if get automated) are usually very flaky and are always slow, that is the problem.

Flaky tests are mostly meaningless. Slow tests are burden.

Reply on news.ycombinator.com