Ask HN: Can you do TDD in a complex product (data science, AI, etc)?

I like TDD, it's pretty useful in many cases, but... you may be starting this the wrong way around.

Your question is "should I use this testing technique?". A better question to start with is "what am I trying to achieve with testing?" Once you know your goal you can find a relevant testing technique to achieve it. Your goals will likely be different for different parts of your product and codebase.

Notice this means different organizations and products (or same product at different times in its lifecycle) will have different forms of testing, because they have different goals and resources.

To help with the decision process of choosing testing goals I've come up with a model, of different goals and the kinds of testing that are useful for each. I'm going to be giving this as a talk at PyCon in May, but have an essay version written up: https://codewithoutrules.com/2017/03/26/why-how-test-softwar...

Since choice of testing strategy is situation-specific it's very hard to give a general answer without understanding the specifics - ping me (itamar at codewithoutrules dot com) if you'd like to talk over your specific situation.

Here is a good link from Robert C Martin (author of clean code) for the question : How doeas the presence of neural nets impact the deve lifecycle in Agile environment http://blog.cleancoder.com/uncle-bob/2017/03/16/DrCalvin.htm...

I generally write tests for the initial/main functionality up-front - usually just a simple call to some func that's currently just stubbed, assert that it does the right thing (e.g. "I want to call my new code like this... and get back this..."). Then implement enough code to make that pass, and perhaps modify/expand the test during that process (exercising edge cases perhaps, or adding more functionality - depends on the project).

Then I might write a whole bunch of related code, fleshing out other parts which rely on the primary part (that I know works well enough for now), perhaps without writing tests first. But if I do that, I will then go back and write more tests until I get 100% coverage later. Undoubtedly you will find corner cases that you didn't cater for when you exercise the code in tests.

It's not done until all my code has 100% coverage, and all the main entry points, important data structs, tricky bits, etc, have documentation of some kind. But I might implement a fair chunk of the code before implementing full tests.

So: some tests up-front (because it makes coding easier), and the remainder of the tests (for full coverage) have higher priority than writing documentation on my list of todos.

In no particular order, I've written non-trivial amounts of Go, Javascript, C#, Python, Java, C/C++, ...

— But your question is not specific enough because the process can vary a lot, depending on what you're coding (e.g. I don't bother writing tests for bash scripts: they usually either work, or they don't), and who / what it is for. e.g. the above process works great if you're building a library (and if you're not building a library, it can help to try and break apart the project into some library-like pieces).

Obviously the above would be somewhat different if I were doing TDD for a web-app, but I'd still apply the same process to the bits and pieces that make up the app behind the scenes - and then have to also consider front-end testing if appropriate.

Not specifically to data science or AI projects, generally test coverage is very important, otherwise how would you ensure that the functionality works as expected and handle error/unexpected conditions?

But TDD approach or writing test cases after the implementation is upto the individual developer/team's preference. Initially TDD approach would be very difficult for freshers or devs who don't have TDD experience, but in the long run, it would help.

In the context of 'my startup' the decision to implement or not implement TDD or microservices or ReactJS is properly made on the basis of business value. Does TDD/microservices/ReactJS make money or improve the odds of survival or provide a competitive advantage or will it create drag on the business?

Good luck.

This is good advice... but it's hard to follow if you don't understand the costs of particular forms of testing, or what benefits they provide. This was part of the motivation for creating the model I referenced in a comment above.

Your earlier question does not make a business case and Markham's answer might be somewhat similar to mine here (unsurprisingly perhaps since I sort of recycle things I read on Hacker News when it comes to startup specific advice and my own experience when it comes ordinary business advice). This sort of falls in the category of both: there is a startup component (survival) and a general business component (YAGNI based cash flow).

The tricky thing about TDD is that Uncle Bob packages it up with morals and ethics and craftsmanship. It's probably a big reason why he is a successful consultant in addition to having coding chops...and I don't disagree with his moral and ethical opinions but they are about writing code rather than running a business and here the important decision is running a business and that's what Markham emphasizes and emphasizes particularly well in his reply to Cooper's response.

I'd put it this way, the first order benefit of TDD is confidence that refactoring will pass fewer regressions to production. But a successful startup does not depend on refactorings because growth drives rearchitecting and pivoting and those mean that many/much/all existing tests may become irrelevant. Riffing a bit, it might be that in a startup the way to determine the benefit of tests is do they get the company to the point where those tests become irrelevant more quickly? Most startups default on technical debt by either running out of money or by throwing out the code base because they acquire different operating parameters due to scale.

Right, so in my model, there's four goals you might have for testing: understanding users, understanding runtime behavior (of your software), correct functionality, stable functionality. Which set you choose depends on business goals.

In a startup environment your goal is often understanding users (does anyone care about this product/feature? is it usable?) followed by correctness. And as you say, stability isn't an immediate concern if you're rewriting everything every two weeks, which is the case for many layers of code.

"I'd put it this way, the first order benefit of TDD is confidence that refactoring will pass fewer regressions to production." - I'd say automated tests, including unit tests and therefore TDD, only give you stable functionality. Not correctness, since only a human can determine that (see essay for fleshed out argument - https://codewithoutrules.com/2017/03/26/why-how-test-softwar..., section on "What means of testing can you use?").

But unit tests (and thus TDD) for code that isn't going to be a stable is actually a negative, because you're trying to enforce stability on something that keeps changing.

--

Could you link to the Cooper/Markham discussion?

Cooper and Markham were at the top of the link in the question. {edit: I have been mistaking you for the OP sandGorgon}

--

I can see how A/B testing might help with understanding users, but to me, A/B testing is orthogonal to Test Driven Development and early on A/B testing would probably not have enough test cases to generate deep statistically significant insights due to small sample sizes and the need to make gross movements rather than fine tuning a working process.

--

For Uncle Bob, TDD is a business process because he is selling consulting at some hourly/weekly/monthly rate. His client base will tend to be looking at improving stable business processes not those taking a spin of startup roulette. For a technology startup, what gets developed in house will tend to be strategic in the sense that it has a big effect on the company's velocity. Getting 'there' in three months is more than eight times as good as getting there in two years.

Ah, thanks, will read through.

A/B testing is orthogonal to TDD, yes, and that's really the point: there's many forms of testing (usability testing, testing user interest, as well as A/B testing) whose goal is completely different than verifying a specification. Often they are useful when creating the specification.

And maintaining stable software is very different than spinning up new product, yes. So you need to start with business goals, work down to testing goals, and then choose testing techniques based on those.

I should add that pure craftsmanship strikes me as a limited point of view: what is the point of "high quality" if no one uses your software?

https://blog.ometer.com/2016/05/04/professional-corner-cutti... has a better take than just "quality all the time" that TDD advocates sometimes take on.

My impression is that Uncle Bob works for clients that have software that people are already using at some meaningful volume (volume rather than scale). I could be wrong.

Well, for TDD specifically I tend to think of it as a testing methodology for libraries. Application logic tends to be hard to test that way since they're about human interaction.

Of course, you want applications built out of libraries. But eventually if you do that you end up at the point of "this is pure UI code and unit tests will do me no good."

So I don't think it's just "stable, mature product" vs. "new product". It's also about code structure and where the abstraction layers are, and which abstraction layers are stable and therefore amenable to automated tests.

A UI can change in ways that a human will not even notice but will break brittle automated tests. There's workarounds, of course, and probably it's possible to build much better tooling than actually exists.