Kent Beck: “I get paid for code that works, not for tests” (2013)

There are other techniques which can give similar confidence, but tests are the easiest one.

How often do you find, despite having written tests, that there is some bug in your software? And how many of those times did you think that you should have considered it beforehand, rather than that it would be impossible to foresee?

In my experience the most useful tests are the ones that came from some unforeseen bug, which was then fixed and a test case built around it, so that it wouldn't get "unfixed".

The least useful tests are the ones for cases you know not to invoke, because they are obvious. Like how you know when you divide by a variable, you know it can't be zero. So you make sure it can't be zero, making the test case a bit moot.

I started my career writing iOS clients, and the obsession with TDD was baffling. 80% of my code was usually either UI or simple Core Data manipulations, while the last 20% was mostly API parsing and a touch of business logic. I wrote a few tests for parsing corner cases or business logic, but they never really gave me any confidence or helped with refactoring, instead taking up time and adding overhead whenever I made changes. I supposed I didn't have enough coverage to get the benefits, but what tests would I write for my UI? What tests would I write for simple Core Data queries (which is assuredly unit tested already)? What tests would I write for my parsing libraries (which are already unit tested)?

Then I started working on the (Python + Flask) API backend, and tests were self-evidently necessary. Python is dynamically typed, which can result in lots of corner cases when doing simple data manipulations. Python is interpreted, which means the compiler/IDE won't warn you about syntax issues without running the code, and you can't catch even the simplest logic errors without running the function. Most importantly, the API's entire job is translating data, inputs are in the API parameters or database, and output is the JSON. It's a perfect function, and tests were obvious. I wrote something like 600 in a week, then used them to make some major refactors with confidence.

What I learned from those juxtapositions was that unit tests and automation are invaluable in certain circumstances. Specifically _any system that creates machine-readable output_ like JSON, populating a database, or even a non-trivial object factory should be unit tested like crazy. Any system that creates human-readable output, like views or changes in an unreachable database (something like an external API or a bank account) needs to be human-tested, there's just no way around it.

Here's my personal thoughts/experiences:

- Testing job is underestimated.

- In General, Developers considered superior to testers.

- What makes Tester position difficult is 'repetitive tasks' . Yes you can automated tasks, but you still need to do some tasks that can't be automated. These are manual & repeated tasks, often boring.

- Some developers are so lazy. for ex: while testing we found 'python syntax issue!'

- Management thinks testers can be replaced once they automated everything. Obviously they push for this.

- I know for sure, there are projects with passionate developers but no-one can really take care of their testing side.

- Dev underestimate/avoid unit-tests & rely on testing team to find basic issues.

When you get out a bit of the IT world, you'll find that people who demand software want to receive something: software. They bought an app, they want the app. Simple as that.

If you are good enough to have your code working without tests, good. If you don't need documentation, good. If you paint your walls with use cases, good. All that doesn't matter, if you deliver the app you were hired for.

And if your app doesn't work... well, everything you've done doesn't matter either. Because you were hired to deliver an app.

Of course tests are good, documentation is good, self-documenting code is good. But only for the IT. For non-IT people who's contracting you (you can be your company, too), he just want the app. The software. Working.

But for those of us who work on a team, it's far more complicated than that, and you have no idea who might be touching your code in the future.

That doesn't means that you should not have them. But you at least should be able to answer that question to be able to evaluate the value that they bring and how much tests do you need and where.

Right now the generated tests are pretty superficial and silly, but the key is that they are randomized. Because of this, we can run millions of variations, some of which will prove to be good tests. Right now I'm hacking an AI that will pick those good instances from many random test runs. If it works, we'll be able to simply take source code and produce (after burning enough CPU cycles) good unit tests for it. This will be a huge help in letting the human programmer only do "enough" test writing -- the AI will take care of the rest. Additionally, the solution can be unleashed on cruft code that no-one dares touch because of a lack of tests and understanding.

(Yes, there will be a business built around this, but that's for next year. :)

- 1. Test infrastructure is too complex. If I have to create a bunch of config files, obey a questionable directory structure, etc. before I can even write my test case, there is a problem. There should be very little magic between you and your test front-end.

- 2. Test infrastructure is too lacking. It is also a problem to have too little support. There should be at least enough consistency between tests that you can take a look at another test and emulate it. There should be clearly-identified tools for common operations such as pattern-excluding "diff", a "diff" that ignores small numerical differences, etc. depending on the purpose.

- 3. Existing tests should not be overly-brittle. Do NOT just "diff" a giant log file (or worse, several files), and call it a day; that means damn near any code change will cause the test to “fail” and create more work. Similarly, make absolutely certain when you develop the test that it can fail properly: temporarily force your failure condition so you know your error-detection logic is sound.

- 4. Tests should not be overly-large. Do not just take some entire product and throw it at your test, creating a half hour of run time and 40,000 possible failure points just because it happens to cover your function under test. It is vital to have a small, focused example.

If your test environment has problems like these, I fully understand the desire to balance time constraints against the hell of dealing with new or existing test cases, and wanting to avoid it completely.

And if you’re in charge of such an environment, you owe it to yourself to devote serious time to fixing the test infrastructure itself.

I think testing trivial code is a waste of time and does nothing but improve coverage numbers.

When you think about a non-trivial problem to write tests, you don't always know what the final code will look like. Maybe you forget an edge case or some small detail in the requirements that will cause you to restructure the code and approach the problem in a different way. In which case, you now need to re-write your tests. You might as well just write tests around the final version.

In some case unit testing is necessary, e.g. for ensuring that a hash function works exactly as defined. However, there are other cases where unit testing is absurd, and black-box API tests or automated tests could do a better job on error coverage. As an example, imagine the Linux kernel filled with unit testing everywhere: plenty unit testing religion fun, but no guarantee of getting anything better, but a risk of new bugs because of the changes and increased code complexity.

Ah, I get it. That explains the piece of s--- I'm looking at right now.

That said, the title might be sensationalist, but I agree with the holistic sentiment of the text.

I've seen dozens of GitHub repos with a "tests/" directory that only contains tests for the constructor and ignores all the parts that should be tested. You don't need to test a constructor, this is stupid. If your constructor is not working none of the other tests will -- BUT HEY, your constructor is working, it is not hard to see it.

When I am writing a module/function, I tend to continuously think of how this can be tested, which helps me design better abstractions.

For example if you're writing a class that uses a socket read/write, when testing you probably need to mock them. If you weren't planning on writing tests then probably you'd have ended by having the methods embedded in the class itself as read/write/close when those methods don't belong to the class and should be in another module called Socket that inherits a Socket interface. Now that you have a socket interface it becomes easier to test your class by passing a mock Socket interface.

> I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. I do tend to make sense of test errors, so I’m extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

On the business side, you don't get paid for code at all. You get paid to make something people want. The fact that you're using programming to do that is inconsequential.

On the tech side, you're not delivering anything unless somebody, somewhere can test it, even if only one time.

So yes, you are getting paid for tests. In fact, that's the only thing you are getting paid for. The nub of the question is what the tests look like and how many you should have.

Now, that doesn't mean you should not test. It means you should understand the limits of unit testing and test what is important as best as you can. Most every software engineering class at universities will cover this in-depth.

> Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.

Indeed, we still "don't know" how to test—more generally, and given the abundance of methodologies and their tendency to go through a hype and dump cycle, I would say we still "don't know" how to write code in the first place.

We'll get there eventually, but for now I would take whichever approach, methodology, tools, and language that I use as having a "best before" date, and invest in it accordingly.

I've long since learned the hard way that if you don't test the functions as you write them, the bugs get buried in the system, and become very hard to find. When you test your code as you write it and modify it (formally on larger projects, informally on smaller ones) this doesn't happen.

That's the advantage of tests, so I can get Beck's point: If the function is so painfully simple that you already know if it's right, (say, an accessor) just by looking, then it's not worth writing a test for it.

Put a sane, normal value test. This will pass unless shit's broken.

Then test edge cases. Test min, max.

Then test some impossible values. If they correctly fail, you pass.

>>I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris).

Is the humility in the parenthetical. There is a difference between arrogance and confidence.

But then again, it may be easier to just set a single round goal like 100% for test coverage. Writing that test for the single expression setter won't cost you a lot.

I've had a small team nicely paid for months only to prove and document the that the product my company was to deliver won't fail in some specific scenarios, specified by the contract.

Those who don't produce the mission critical code (or believe what they produce is not on the critical path) unsurprisingly see the investment in the tests questionable. Of course, there is always a real danger of doing something "just because it is done" even if there's no real need.

If Kent Beck is coding in a private bubble, he can do whatever he wants that makes code work.

But proper syntax is what gets you code that works. You aren't paid directly for it, though. Tests are not a direct path to code that works, but they can be a big help.

Code that "works" doesn't just mean it runs/compiles/passes CI/etc. It has to continuously add value. It can do this by running properly and efficiently across a wide variety of likely or infrequent conditions, as well as some exceptional scenarios. It can do this by being written clearly and not adding technical debt. It can do this by being as simple and/or as replaceable as possible. And ultimately, it can even add a final gasp of value by being easily deletable.

https://www.youtube.com/watch?v=0WMWUP5ZHSY

Which is unfortunately complete opposite of how TDD was interpreted, especially in its glory days (and in some corners, up until the present day).

Nobody needs division by zero tests if there are already guards in place so that can't happen, but it's quite helpful to have a "A goes in, B should come out" view from a client/user perspective. As long as behavior appears correct to the client and is not exploitable you're good to go.

Of course given no formal requirements, all that is possible are tests of the technical implementation. Regression errors will still be inevitable for customers and stakeholders, despite the "programmer" being able to claim his or her tests passed.

We need to find a way to stop kidding ourselves and find a way to test the right thing.

I'm not advocating testing getters/setters but not testing because "I don't write those kinds of bugs" can burn the next junior dev who might.

Testing is as much about finding bugs early as it is making your more agile in the future.

Its all a trade off that needs to be communicated to whoever is paying you.

The problem with comments like this is that they're too ambiguous. Someone who doesn't want to spend the time to write unit tests, will use this ambiguity as a mechanism for justifying their laziness.

That's one problem with the TDD mindset. If you start by looking for things to test, you might come up with unlikely scenarios or cases that don't matter much for your user.

I agree with Kent Beck mostly. I would add that tests can also be used to maintain invariants for future changes to increase maintainability. I just hope this quote isn't taken out of context.

It really depends on what your project is, what your goals for maintainability are and what programming language you use.

Two things about testing: - test to confirm your spec - if you have trouble writing tests, your design is probably flawed

a) Bad team mates. b) Future developers.

I've been burned one too many times with junior devs making cavalier changes in code they don't understand. Unit tests were THE solution for catching these changes.

Most comments seem to equate:

  "regression tests"=="TDD"

  "regression tests" is subset of "TDD"

1) the ergonomics & design of the API you're building by way of writing the tests first. In this sense, the buzzword acronym could have been EDD (Ergonomics Driven Development). Writing the usage of the API first to see how the interface feels to subsequent programmers. Arguably, a lot of incoherent/inconsistent APIs out there could have benefitted from a little TDD (e.g. func1(src, dst) doesn't match func2(dst, src))

2) a sort of specification of behavior by usage examples ... again by writing the tests first. Consider the case of programmers trying to figure out how an unfamiliar function actually works. Let's say a newbie Javascript programmer wants to know how to use .IndexOf()[1] What do many programmers do? They skip all the intro paragraphs and just hit PageDown repeatedly until they get to the section subtitled as "EXAMPLES". With TDD, instead of examples being relegated to code comments "sqrt(64) // should print 8" , it formally encodes the "should print 8" into real syntax that's understood by the automated test tools. (Test unit frameworks typically use the keyword "Expect()" as the syntax.)

3) an IDE that's "TDD aware" because it creates a quick visual feedback loop (the code that's "red" turns to "green") during initial editing. The TDD "artifacts" can also act as a "dashboard" for subsequent automated builds alerting you that something broke.

So TDD is a "workflow" and from that, you address 3 areas: (1) design (2) documentation (3) quality assurance via regression tests. With that background, the original Stackoverflow question makes more sense: how many "test cases" do I write because it looks like I can get bogged down in the test case phase?!?

[1]https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

That's like saying "I get paid for functioning software, not writing code."

Yes, you get paid for the output of the act, not the act itself.

Such as?

When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

Allow me to explain:

Production use of code IS testing (manual, etc). Because it is an observation of the system state.

All of the world is testing. Every system is inherently a quantum mechanical one (ie: the observer is constantly testing the state of various systems to ascertain some level of confidence)

If you are going to test anything... then you should test the Use Cases (ie: Interactor objects). Don't have Use Case/Interactor objects that encapsulate intent? Well, you better understand it since the world, and therefore software, is all about intent.