TDD from the Factorio Team
factorio.com
factorio.com
> This is the beautiful thing about having a company that isn't on the stock market. Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that. Luckily, we don't have any shareholders, and we understand the vital importance of this investment in the long run. Not only in the project, but also in our skill and knowledge, so we do better next time.
This is reassuring the notion of what I think actually matters, what the real essence is of developing a product, may that be a piece of art and entertainment (like here) or a productivity tool etc.
There are creators and there are consumers. We split them up by developers, designers, domain experts and so on, but what matters is that all the other participants, especially those who can exert power traditionally are not part of the essence and if not being careful and responsible, can easily add complexity and limitations that are entirely accidental and can even be harmful.
This reminds me of the agile manifesto, modern UX approaches and other processes that are driven by creators, but are often and very unfortunately being bent over backwards to fit into hierarchical power structures.
> TDD actually is the constant fast swithing between extending the tests and making them pass continously. So as you write tests, you write code to satisfy them basically at the same time. This allows you to instantly test what you write, and mainly use tests as specifiation of what the code should acctually do, which guides the thought process to make you think about where you are headed to, and to write code that is more structured and testable from the very beginning.
The important notion here is that TDD is not about tests and correctness, but about development. It continuously checks assumptions and explores the surrounding code, state and data until a sufficient solution is found.
If we squint a little we can see how closely related TDD with REPL Driven Development is. In essence it is the same thing and even has similar results, where the tests or REPL code can be left as an artifact for further, likely historical understanding.
We know now that neither is sufficient for a high degree of correctness, but they are certainly useful for understanding and development.
Yup, writing tests helps me sleep at night. TDD helps me manage my mental resources and iterate.
(another reason is communication--we code for our colleagues first, then for the machine: https://sonnet.io/posts/code-sober-debug-drunk/)
IIRC smalltalk had a workflow where you'd debug and write your program at the same time. You'd just reach a path that has not been implemented yet, break, implement it and continue.
This has been implemented to varying degrees of reliability, granularity (in terms of preserving state) and utility. It's a common concept but I don't know the name for it!
Examples:
https://en.wikipedia.org/wiki/Continuation
https://en.wikipedia.org/wiki/Fault_tolerance
Maybe fault tolerance is the right word? But it doesn't imply that you actually modify and build the thing while it already runs.
a good read is ROC https://en.m.wikipedia.org/wiki/Recovery-oriented_computing
An example of fail open design in the real world, is electronic magnetic locks at apartment complexes or business buildings. If the building loses power the doors demagnetize and unlock, then indefinitely remains open but failing because it is always open now. The opposite would be the building loses power and the doors fail close and lock the doors indefinitely when power is lost, and now people can’t escape.
This is exactly what I do with the PyCharm - hit a breakpoint, write the code as it should be, and execute it in the debug REPL to do basic initial testing [repeat]. Extremely productive.
https://gist.github.com/daveray/1441520
Even if you don’t follow along and try it, you can probably get the gist of how experimental / exploratory you can be.
rev([1], [1])
rev([1, 2, 3], [3, 2, 1])
rev([1, 2], [1, 2])
rev([], [1])
rev([1, 1], [1])
But if you look at logic programming with more expressive systems you can have something like what you propose. We describe what we expect to have and the system deduces a result. Not quite what you want but its closer.
Now there is also an ubiquitous logic system that many use: static typing. In a sense you are describing the general properties of something and the compiler infers optimizations based on your assertions. The concrete program is not a line by line translation from your code to machine code, but perhaps looked at in its entirety.
I agree that there is a lot of merit in pushing these things further and further. Right now we're kind of in a stage of patching things together. But I hope and assume that programming becomes more holistic in the future. Ironically we have to look at the past first, there was a lot of momentum in this direction up until the 80's roughly.
Beware that machine learning in a single pass/fail is more like having an infinite number of monkeys trying to write the works of Isaac Asimov.
[Edit/Update] All of the tests could be individual values, so non-zero (but nowhere near all ones) might help. Thanks for making me reconsider this, sdenton4.
Maybe you could move the goal/fitness function along the way. So start with something that compiles. Then having the desired input output. Etc.
Each 'bandit' is a random boolean outcome, governed by some hidden success probability. Thompson Sampling trades of exploration and exploitation. If there's no successes ever, then all bandits are equally bad, and you just keep exploring randomly until you find some success (or give up). If you do have some success, you can try to exploit it.
For a problem with continuous parameters, you can discretize the parameter space by binning, and then choose randomly within a bin for each trial. 'Exploiting' a particular bin might lead to breaking it into more bins for finer resolution.
Aside from that, to produce a program that did what is actually required would require test cases and functions that cover the set of inputs and outputs. This is trivial for mathematical functions, but impractical (or impossible) for more general applications (e.g. anything dealing with human inputs).
This doesn't become practical until you can do it on a quantum computer with millions of cubits.
We are advancing, but the current state is... not mind-blowing yet (albeit somewhat cool!). See [1] for an example interactive demo and [2] for the corresponding presentation.
Still a decade off from production, though. That it doesn't just take "test cases": you have to know how to formally express the program properties, which is a separate skill from both unit testing and implementing.
TDD is where you both write what you want, and you do the implementation also.
Show examples of French retirement pension computation, and watch if a computer can actually commit petit-suicide.
I have (seriously!) never heard it reference like this at all. Every introductory article, every book, every tutorial, every video about TDD (or BDD) that I consumed, emphasised that it is about "workflow", about "a way of development" and not about percieved correctness.
Do note that the crucial different is that TDD > automated testing. In the sense that with TDD you write automated tests (that live in a test suite). But that "writing tests" or "developing a testsuite" is not nessecarily TDD.
In my personal opinion this claim goes too far. I think we can often explore aspects of design through code (including tests) but it can't be the main driver of design in any circumstance, nor does it by default or in general make our modules and programs more correct.
The key benefit of doing this is that we have to think of code in terms of invariants and testability. But that is just one of many, sometimes conflicting factors and there are certainly more approaches to doing this than writing automated (unit-) tests.
The TDD book itself is barely even about testing, it's almost entirely about development processes and different types of refactoring and how to use tests to get from a to b.
Correctness is rarely emphasized if at all.
Right, in TDD there is "always" a range of values against which you are not testing. A developer can only create a finite amount of tests, and the program will fail with any of the missing permutations.
I took to TDD pretty easily because I was already used to doing short run-it-and-see-if-it-works cycles. The main difference was that instead of checking via eyeball, I talk the computer to check it. This is slightly slower early on, but so much faster once a program is big enough that manually checking everything would take a while.
> that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies
Even in a world without shareholders, we still are building things for users. 6 months without improvements has an effect on them, too. When possible, I think it's better to spread cleanup work out. Even if one spends 80% of the time on cleanup and 20% on features, that is much better for relationships than going dark for a quarter or two. And in my experience, continuing to do productive work during that period makes the behind-the-scenes improvements better.
They did exactly what you suggested! The relationship with the users never went dark. They kept up with bug fixes, and kept feeding new features at a more than acceptable pace. [Edit: Here's the pace of their status updates, as evidence at https://www.reddit.com/r/factorio/?f=flair_name%3A%22FFF%22 ]
I didn't quite believe that Uncle Bob's lessons worked in the real world, but if the Wube team is sold on them, that's about as good as an endorsement as I'll ever get.
I keep thinking I should try Factorio. But given amount of time friends have spent on it, I'm scared I'll never be heard from again. One of my friends has built a whole Grafana setup for it.
As long as you're immediately understanding, you'll continue to have fun.
As soon as you need to stop and research to understand what it is that you need to do next, and then start to re-architect a bunch of things, you may or may not realise this is the same thing you do for a living.
Nothing quite like burning the candle at both ends, for fun and profit.
Personally, it's not for me. Others (including some of my colleagues) definitely do not share that sentiment.
That said, I enjoyed Mindustry for a similar period too. Factorio-lite tower-defence game.
TDD is often sold as a fix-all solution, which is incredibly appealing to mgmt and quite fun for most programmers as a new paradigm, allowing for quick adoption.
It also has its uses, especially in the enterprise space where requirements aren't often clear. But I don't know many good programmers that truly stick to the dogma after the honeymoon period. It becomes just another tool in your toolset.
Uncle bob is a salesman, not a "craftsman".
So the effort you put into writing a test for a bug, has most likely a negative return on investment. That time could be better spend somewhere else.
- Writing a test for a bug is part of understanding the bug, much like rubber ducking, it helps you make sure you know exactly what causes the bug.
- The better you are at writing tests, the less time it takes you to do so.
- Having tests for the code means the code is testable. In general (but not always), code being testable means it's better code (more readable, less complex, etc).
- Having tests for code means you have to worry less about breaking it when making changes. This allows work to proceed faster.
So it's not just "did adding this test prevent this one bug from re-occurring", it's "did adding tests improve our development overall". The later is far more likely to be true than the former.
When your problems are much more fundamental, like teams just flat out don’t talk, writing the test is an easier investment.
Understanding, reproducing and fixing the problem is the hard part, once that is done making a test out it hardly takes any time.
Writing the test does not only prevent regressions, it also helps with confirming you actually fixed the bug.
It all comes down to return on investment. For example, I used to agree with TDD that for every bug, first write a test that fails, and then fix the bug. That way you prevent regression.
So I proposed this to my manager, and he responded: we tracked all bugs in our system for the last 10 years, and when you look at fixed bugs that get broken again, it only occured very rarely. So in the end, doing that was not the best investment of effort.
The benefit is basically all the tests that you don't have to change and give you confidence that your refactorings don't break certain things.
But the drawback is all those tests that do need to be rewritten because of the refactoring, and so will slow you down again.
But in the end for this use-case, I think it's a good ROI. Probably the best use-case for TDD.
Look at the concept of sociable tests.
Maybe if you work in a monolith, sure. But most of us work in distributed systems with really complex behavior. No TDD suite in the world is going to catch a thread pool issue that’ll open a circuit breaker in your client.
I don't know your setup. But this is normally exactly what system tests do (aka E2E tests, GUI tests and so on; the tip of the Testing Pyramid). In distributed systems, those often live outside of the various components' codebases; maybe in their own project even.
Edit: because it is a) very impractical to check these things manually, b) often simply impossible to check them before each release and c) requires hoops and tricks to get in a state where such events/regressions occur in the first place; a state that is near impossible to get to manually.
I get the feeling that if more people were okay with developing monoliths (albeit modular ones), then a lot of things could be easier to do, such as doing TDD properly and being able to refactor without fearing the unforseen consequences.
Heck, maybe the projects that i work on in my $dayjob would even have proper documentation, decent test coverage, as well as a set of tools to support the development processes, instead of having to waste time managing the configuration, deployments, as well as system integrations. Maybe it's a matter of there not being enough workforce (or the managerial folk not seeing much point in investing resources into testing and other ops related activities that don't generate business value directly, a worrying trend i've noticed), but right now i'm implementing Ansible and containerization to at least simplify some of this stuff, but it feels like an uphill battle, as i'm also supposed to ship new features.
Surely i'm not the only one that kind of sees why the person that you're replying to would express the viewpoints that they did? It's hard to do elaborate E2E tests when everything is figuratively on fire constantly and you're struggling with a complicated architecture which may or may not be necessary. I'm probably projecting here, though, but every single enterprise project that i've worked on has been like that.
I was working on a new project recently. Over the first month or so I went through four or five significant iterations of the architecture before I settled on one that seemed to have a healthy mix of power, flexibility and simplicity.
Each time, I found the test cases I’d identified during earlier iterations helpful. Ultimately they led me to a more rigorous analysis to make sure I’d covered all required cases in all required places.
However, I hardly ever kept the test code from one iteration to the next. Each unit test follows a general pattern of arranging whatever scenario I want to test, then calling the function under test itself, and then asserting some expected result. With a significant architectural change, the interfaces for arranging things or to the function under test might be changed, leaving much of the code in the test obsolete. The responsibility being tested might not even be in the same part of the code any more, meaning a whole set of test cases now need to be applied to a different component in the system.
In my opinion, as soon as a test suite finds a bug it has added a lot of value, even if it's rare.
I have often observed an evolutionary behavior on tests: tests that pass easily survive
That sounds very hand wavy.
It is making the assumption that:
- your bug tracking system and people are so good that they have found all the duplicate tickets.
- they don't make mistakes and find all duplicates for tickets.
- your code is so good that it hasn't had any side effects that caused regressions.
- your boss is so good that he has a full grasp on 10 years worth of bugs.
In return, I get to have some extra confidence that a bug doesn't return (which would be embarrassing, even if it's infrequent!) and I get a more thorough test suite that lets me refactor more quickly and aggressively. And if an old bug does come up again, the advantage is not only that the test will catch it before a release, but also that the person fixing the bug won't have to go through the effort of figuring out how to reproduce it from scratch—it's reproduced right in the test suite!
So I am not sure that just looking at how often regressions actually happen in the existing codebase is sufficient to make any real conclusion by itself.
Once you understand how to design testable code, tdd offers minimal benefits.
The real value comes from thorough, automated testing suites, whatever their origin is.
For things like reproducing bugs, in particular, functional tests plus a debugger have been indispensable in seeing how the code actually behaves.
Sure, production monitoring is nice. But it really only tells you about high level problems, and there are entire classes of bugs, eg. silently corrupted data, which you'll never see there.
Unit tests' inability to handle state is way too often viewed as a problem with the code it can't properly test than the general crappiness of this form of test.
For some reason "code testability" is almost exclusively used to refer to the overuse of dependency inversion to make it easier (although not necessarily more useful) to write unit tests, though.
I find that code falls into three camps - integration/logical/mixed.
If it's integration code only an integration test is really useful and a unit test is a pointless waste of time that will do little more than mirror the code you wrote and fail when it is changed for any reason at all.
If it's logical code a unit test is most useful and integration tests are likely going to be too slow.
If it's mixed, integration tests are most useful although drawing out and decoupling the logical from the integration code and testing them separately is better in the long run.
Our projects are so database centric, I've almost never seen a true unit test fail. The majority of time it's the tests that use the in memory RDBMS.
Automated testing is useful, for sure. However, TDD purists tend to focus on one very specific type of testing: automated unit testing, in the small, where you already know the expected output for given input and can easily specify that output using assertions in code. By its nature, TDD emphasises being testable in that specific sense above all else. I don’t think that is necessarily a good thing, partly because that type of universal unit testing may not be a good strategy for every software system, and partly because other useful properties of the code might be diminished because of the changes needed to make it “testable” in the TDD sense.
Well, "knowing" (or manually generating) the output for your code is required to check whether your code is working. If you execute your code and copypaste its output blindly, you risk testing the wrong thing, don't you.
And if you can't specify your code's output, how do you test?
BTW, I agree with the fact that TDD is overly pedantic at times.
Not all code is written knowing the answer in advance. Sometimes you’re implementing a mathematical model to predict next week’s weather, or the effects of a change in economic policy, or the performance of different hull shapes for a ship under different conditions at sea.
My second standard challenge to TDD evangelists is to write a program to draw a classic multicoloured Mandelbrot image, showing an arbitrary rectangular region of the complex plane, using only the mathematical specification without any reference to existing code or output data.
This can be done with a short program that is simple enough to verify by eye that it matches the required mathematical specification. However, the problem has an infinite input space, its fractal nature means predicting output values is impossible in the most interesting parts of that input space, and you have no idea where those interesting parts are anyway.
I guess that'd still be TDD, but with the first T standing for Theorem :p
1. How do you choose which specific cases to calculate manually?
Remember, you have no prior knowledge of what the correct answer looks like or where the interesting parts of the input space are. You have no way to determine whether any given set of manual calculations is representative of the work your program will be doing or covers the areas with the greatest risk of error.
2. How practical is it to make all those manual calculations?
Even in this very simple case, calculating the correct answer for a single input point manually might require hundreds of complex arithmetic operations to be performed. That’s going to be slow and error-prone. After all, isn’t that why we’re writing a program to do this for us?
Now, how does this idea scale? What if our program isn’t computing a nice analytical solution to a simple arithmetic problem, but instead running a complicated numerical method to process many thousands of data points in each input? It quickly becomes impractical to rely on this strategy for testing.
3. How will having a set of known outputs you can test against drive an implementation from scratch?
I mentioned before that Mandebrot is my second standard challenge to TDD evangelists. The first is to write add(x,y) driven by tests. After a bit of back and forth, this invariably ends up with an implementation that was generalised from however many specific cases were given to the general case. Invariably, that generalisation is the step that actually creates a useful solution to the original problem, and invariably it uses an insight that was not driven by the tests.
Our Mandelbrot scenario is the same situation, just a slightly more complicated example. No matter how many individual tests you create by choosing sample points in the input space and manually calculating the expected output, you won’t have a systematic way to work back from those answers to derive a correct general implementation of the Mandelbrot calculation. Your specific cases might be useful for verifying an existing implementation, but they give you no insight into how to write a good implementation from scratch. (If I’m dealing with a particularly strident advocate of TDD, this is the point where I mention the word “sudoku”.)
And again, we have to ask how this process scales. What if we had a more challenging problem, say extracting an audio track from a video file and running a speech recognition process on it to generate subtitles? It might actually be easier in that scenario to identify individual test cases: just take a known video file as input and write down the expected output for it, and now you have an end-to-end test. However, it’s still true that no amount of end-to-end tests will necessarily offer any insight into how to structure a good implementation of the required functionality in detail, nor will it tell us how to implement any specific part or generate useful unit tests cases for those parts. End-to-end test cases might help us to verify an existing implementation, but in general they won’t reliably drive a correct one from scratch.
You can’t reach a program that solves the general case by iteratively writing a failing test for a specific case, making the smallest change required to make that test pass, refactoring, and then moving on to the next test. The important step — what you called “fully implementing the add function” — is taken by implementing some other insight that is not driven by the tests. That step isn’t a refactoring either, because it very much does change the behaviour of the program.
To design and implement a good program, sometimes you just have to know what you’re doing. There is no substitute for understanding the problem you are trying to solve and how you intend to solve it. And if you have that understanding and you design and implement your program accordingly, what is the value of following the red-green-refactor process compared to simply writing any unit tests you find helpful for verifying your implementation?
For me, unit testing (in the typical xunit style associated with TDD) is most useful for basic data processing code with predictable outputs. That might include a wide range of code, from little utility functions on strings to the business rules for whole CRUD applications.
On the other hand, anything with input or output data in an awkward format, anything communicating with any external equipment or remote API, anything involving nondeterminism or heuristics or where the purpose of the code is to perform some calculation where you don’t know the correct answer in advance, these kinds of code don’t tend to fit well with a TDD approach and that style of unit testing as the main test strategy IMHO. Those cover a pretty wide range of code as well.
If you don't know about what the code is meant to do (non-determinism, heuristics, etc), to me that you're targeting the wrong level of abstraction. At some level you know what it's meant to do, unless you're working on an abstract art project.
TDD done dogmatically is a mess, sure, but then so is anything.
• xUnit-style tests for specific cases
• Property-based testing
• Snapshot-based testing
• Manual testing
• Formal verification
• Peer code review
All of these can be useful under the right circumstances.
If your code produces output that is best checked by human inspection but shouldn’t then change (or change very much) then snapshots may be a good choice.
If your code won’t deterministically produce the same correct output every run but whatever output it does produce should always satisfy certain conditions, maybe property-based testing is a good way to go.
If your code involves communication with external equipment that requires operator interaction to do anything interesting, fully automated testing might simply not be possible. In that case, manual integration testing with someone physically operating the equipment might be appropriate.
Formal verification covers a wide range of possibilities from the likes of basic static type checking all the way up to the use of automated theorem provers in specialised programming languages. It almost always has some extra cost in terms of annotating the code but it can sometimes produce far more powerful evidence of correctness in the general case than any test suite checking individual cases ever could.
Code reviews are pretty much universally good, as long as you’ve got enough people available with the relevant knowledge to do them.
Sometimes, these testing techniques are complementary and using more than one of them together might be beneficial. At other times, a coding style or process that favours one might make another more difficult.
This HN discussion is mainly about the style of “testable” code that TDD tends to produce, with many small units and lots of dependency injection, which is of course very friendly to small-scale unit testing. However, it might also be more difficult to review because of all the configurability and indirection. If it relies on doubles to stand in for external resources, it can end up testing the accuracy of the simulation more than anything else, so adding very little (justified) confidence that the real system is operating correctly. And as a rule of thumb, individual testing of specific cases may be less effective than testing large numbers of generated cases with property testing, which in turn may be less effective than proving that all cases work via rigorous analysis.
And this brings me back to where I came in, which was that prioritising “testable” code in the TDD sense isn’t necessarily a good thing. The real meaning of “testable” depends greatly on what type of code you’re writing and which testing strategies are most helpful.
They do already do this (hence the blogpost), but it's something kovarex hasn't explored before, so they're pretty new to TDD.
Thats just the thing with TDD: used properly (which is clearly tricky for many of us), it should help you think through what the inputs and outputs of the program you hope to write are. It does assume you know what you want to build though, even if you're unsure of how to build it.
I think of it as a design tool to be honest. When I'm not quite sure exactly how something should work, but I have a clear idea of what I want it to do, tests help me work through it in a way that produces extensible code.
If you have a problem that is well understood, well documented and that has a straightforward solution, TDD is great because you know where you are going.
But a lot of the time, you have none of that. You don't know where you are going. You need to experiment, you need to iterate, you need to occasionally change directions completely. When you're doing that, TDD holds you back and gives you nothing.
Rarely are we building something with absolutely no idea what the final outcome should be, are we? There’s a general idea of, “this new feature should mostly work like this” description, correct?
If not, you’re doing code very different than the stuff I build.
And that might be the case! But I’m curious what kind of projects you’re working on where you have basically no idea what you’re aiming for.
Even when I don’t understand what’s wrong, I’ll generally have an idea of either: * what is undesired behavior to fix, or * new behavior to add
I can at least play around with the code a little to see how that impacts existing tests, or write some simple test that will demonstrate a really high level desired outcome. As I iterate the functionality, I might see the opportunity for another assertion or three, and then keep iterating on that until I’ve figured out what I truly want, both functionally and test-wise.
Once I know, I can write a test based on just that information. The problems usually come well after that step as my implementation becomes more complex with the size of the app.
I have done a variant of TDD where the output is slightly uncertain where:
* I build a test with everything filled in except output of some kind.
* I write code that generates an output I want to eyeball.
* I run the test in "rewrite mode" where the output is generated by the program and saved with the test. I eyeball it to check if it's ok and then commit.
* In CI it checks the output against the fixed version.
There's plenty of scenarios where it doesnt work or is generally inadvisable but I find it to be a super effective technique where it does.
I think it's an approach that would work better for code that generates text, images, sounds, than common-o-garden TDD.
If you are learning a new language, new paradigms or new frameworks, then, indeed, often testing is an extra burden that adds little at this point. But in this stage, I hope, one should be aware that the code you are writing is bad anyway. In a year, you'll probably be very ashamed of all the things you did there.
But in languages or paradigms that you are already familiar with, TDD helps a lot when learning new things. You can very easily learn about a new library by writing tests that use the lib. Or learn a new pattern by implementing them in tests.
For example, write some tests that wrap a Stripe library, then change the test to change some setting, call a method, or feed it some weird data, and assert some outcome rather than manually putting stuff in a cart, filling your address, CC details and then see that the setting you thought did X actually does Y. And repeat again.
Would you feel that way about the linux kernel for example?
Indirection can also make it harder to rigorously analyse all possible paths through a piece of code and make sure you’ve handled all required cases.
I used to work on a library that solved a certain type of mathematical problem, written mainly in C++. It was built separately for many target platforms in production: different operating systems, processor architectures, compilers. Each supported combination of these might require special provisions in the code due to compiler bugs, hardware limitations, etc. Then you’d have diagnostic code that was only included in the debug builds used during development. There are more than enough preprocessor shenanigans to go around in that kind of environment already, and another layer for test-related flags and checks isn’t going to break any records for most readable code.
I think this is just internalized pain of working with C++ (I suffered a lot with that 20 years ago). For systems-level stuff, I'm getting very excited about Zig, I imagine it will be basically "equally easy" to instrument these sorts of things at compile-time in Zig, as types are first-class values at compile-time. Zig is also super-composable; I have an example prime sieve that can run 60+ settings (combinations of single threaded, multithreaded, bool array, bitmap, hyperthread-awareness) by using a compile-time for loop across a series of instrumented settings; so long as I stay within the standard library the language itself already takes care of the polyfill for platform stuff, and I have access to those flags, if, say I wanted hyper-optimize for things like sse. I don't see any mocking libraries yet, but I doubt that instrumenting a mocking namespace in test will be any more difficult than it would be, in, say, elixir.
I can assure you the linux kernel is not TDD'ed.
For example, we might see ways of expressing indirection for testing that then get removed when building the final product. Or we can look at the great strides made in virtualization since the Linux kernel was started. Something previously hard to test becomes much easier. And I'd expect more so if the testing had co-evolved.
Of course, there still might be hard-to-test areas. But given the massive ingenuity applied to the kernel development process over the years, I expect that they would have advanced what's possible in TDD if they had pursued it.
If indeed they have continued for decades to do it "the wrong way", imposing costs onto the project and society proportional to the number of preventable bugs creeping in, is this not a major scandal for the industry?
I suspect that the core team have reasons that they have not developed tooling to enable TDD of the Linux kernel. And that those reasons are more substantive than "meh".
This is a language/library thing. Some language/libraries let you do TDD with very little indirection, a table of mocked dependencies as compile-time configuration, and all of the mocking events happen in code in your tests.
This is another underexposed feature of TDD: reviews. When I see a PR with solid test coverage, and those are green, it greatly speeds up the process. I just need to read through tests, see if they make sense, match the requirements and test those. And then glance over the added code quickly.
It (unfortunately, I might add) is not always possible to flat out reject a PR without solid test coverage, so it often is less work for me to add some test myself to a PR, than to manually poke around the changed code or to read through it in detail. This is not TDD, but testing-after-the-fact, and those test often are of much lower quality, but at least this is less work, for me.
Testing greatly speeds up the work for a reviewer.
But consider this article on how it is tested: https://embeddedbits.org/how-is-the-linux-kernel-tested/
It mentions a lot of automated testing, which is good. And if you're doing automated testing anyhow, I think TDD is the most effective way to get there. But consider this:
"The kernel development process also proves that there is a big focus on testing since around 20% of the development process (2 weeks) is reserved for code integration and the other 80% (8 to 10 weeks) is focused on testing and bug fixes."
80% is a lot! And that also suggests significant lag between writing a bug and having to fix it, which means a lot more time figuring out what went wrong, and a lot more code written after the bug, meaning really fixing it might be expensive.
So I think it's reasonable to ask how much more effective kernel development would be if they could safely eliminate that 80% of time spent on cleaning up after the 20%.
I've worked at a TDD-focused company and it always felt like I was coding through molasses. Some of us weren't as strict about TDD and I didn't notice a difference in our code quality.
To me, doing TDD right is about having the minimum amount of testing needed to keep bug rates extremely low. It's also about keeping the test code as well factored as the production code, such that changing or refactoring the production code doesn't create disproportionate work in the test suite to bring it back in line.
An experiment I've tried repeatedly is to start a new project and shoot for zero bugs. I think I'm pretty smart, so that at first seems achievable without tests. But pretty quickly complexity increases that I at least have to manually test. And not long after that, manually testing everything gets tedious, so I have to start automating the things I am manually checking. If I keep pursuing zero bugs and feeling like I'm spending my time optimally, I keep ending back up at TDD. Maybe try that for yourself on a hobby project? You might find that the TDD-focused company was better at talking about TDD than doing it.
TDD can be nice for small modules when you have a good idea of where you want to go with your code. For large modules with a lot of unknowns it can be a pain, and in those cases I prefer testing after a first draft.
Perhaps you haven't built a module that was very large yet? Maybe try writing a large module with lots of unknowns. You might find that TDD gets in your way more than it helps in the early stages.
As to whether or not I have enough experience with TDD to judge it, I'll leave that up to you. But I've been doing TDD ~20 years, and I wrote a lot of code before that. Let me know if that's sufficient.
When you make a change to Guest, you only need to change the GuestObjectMother and any tests that may directly be involved in the change to Guest. Tests that simply require a Guest to be involved in the test probably won't have to change, as they're simply retrieving the appropriate Guest object and then handing it off to something else.
For example, I think of the experience doing TDD in Rails with all its ORM and callback interconectedness to TDD on functional code that is mostly exchanging values between systems. All the stub and mock crap goes away since values become the boundary of the system. Gary Bernhardt has an awesome talk making that exact point.
I'm now of the opinion that if TDD is a struggle, its a sign that your environment has serious underlying flaws.
That's generally why I think functional programming is taking off. It's much easier to reason about input-output of stateless functions. They're much easier to test and develop. They're way easier to understand for the reader, because the reader can understand what's happening in one place, instead of having to look through many different files, creating a complex model of all the inheritance relationships between classes.
I consider the 50-line function to be "less ugly" than the same function butchered for "testability" into a bog of poor abstractions and single-use smaller functions.
Unfortunately this approach tends to be less effective in languages that do not have good support for functional ideas like Java and C#.
That's because it is
I pointed out the 10 year old Norvig vs Ron Jeffries fight[1] which demonstrates that TDD is useless when you don't already know what you're writing, but they just looked confused.
The other problem here is that unit tests never break (since you've mocked everything that can break) and therefore aren't worth running; it might be more productive to write them for TDD and then just not commit them.
As posters in your linked thread point out, the irony is that Norvig would have probably been better off using TDD since he knew the general shape of his solution already where Jeffries likely should have been writing several spikes to explore the solution space rather than using TDD since he was unfamiliar with the problem and did not yet have a good solution.
Of course the 80%/20% figures are estimates. The longer your code is maintained the more likely it is some tests will break.
Upgraded to NodeJS 12 where some details about sorting arrays were changed, and we got 2 failing tests. Before that, I would have bet money that those tests would never fail.
Now that they've failed once they've become regression tests (one of the most useful kind of test), but if you set out wanting to test the platform under you I think you'd want to do a lot more work than that.
The point of our tests was to document a requirement we had - in this case, how some data sorted. That test was able to eliminate questions of the "how" - whether the sort was done in our database layer or our NodeJS layer - and let us focus on the important outcome: are the things sorted in the right way?
We didn't write them because of the NodeJS 12 upgrade. We wrote them 5 years ago. We added them because writing them captured a requirement. And when an unexpected reason caused that functionality to break, we avoided shipping a bug in our software.
Regression tests are certainly useful, and every time we ship a bug, we attempt to write an automated regression test. But I wouldn't argue they're one of the most useful - equally useful is any test that captures a requirement.
That's fine for code that never changes, or that has no logic in it. As soon as you want to change some code, you'll want a portion of the tests to require modification and a portion to stay passing.
Gives you something to compare against if the dependency changes its behaviour
I learned TDD ~20 years ago from Beck's "TDD by Example" book. For me it's far more than "just another tool". I'll certainly do exploratory work with throwaway code. And in the early stages of something, I might be a bit slack. But the lesson over and over for me is that if I don't get to significant test coverage soon, I end up wasting a lot of time on bugs. And TDD is the easiest way to get to test coverage. Test-first ends up feeling like a set of small successes with the red-great-refactor loop. With test-after, going back and adding tests feels more like drudgery, and I'm less likely to have written the code in a testable way.
So TDD is not dogma for me, just the inevitable place I end up if I want to maximize the amount of time getting things done while working on a non-trivial codebase.
1.) I'd much rather lean on the type system to prove things than automated tests if possible. But of course depending on the language that often isn't possible.
2.) I find that only a small fraction of my code really benefits from automated testing. This is the logic/calculation parts of the code. The rest is slinging IO and SQL queries which all ends up being mocked out anyways and the tests just become secondary implementations of the original code.
That said, my solution to that isn't to not test, it's to start with broad, high-level tests that make sure the whole thing works together. Then I add automated tests whenever I get bugs, whenever I find myself manually testing something to make sure it works, and whenever tests would have documentary value.
For those edge-of-system things, if you haven't tried it you might consider abstracting them out. E.g., with SQL persistence I might end up with an abstract Store and implementations of InMemoryStore and DatabaseStore. The fast version of the tests uses the former and I run it all the time when testing. The slow version uses the latter and it gets run less often. That can get rid of a lot of essentially duplicative testing while still giving a fast feedback loop.
And it's not without drawbacks. Increased development time and the needs to maintain the unit tests are costly, but rewarding.
Also I don't like too much interface and mocking only for the sake of testing. I find it usually breaks when integrated and makes code harder to maintain. Maybe I'm just inexperienced.
An important idea of TDD is that it allows you to discover those "unfulfilled points" in the tests. When writing code (the tests) that use an API, instead of when writing the actual API.
When writing code that uses objects, methods, interfaces and so on, you are in a mindset of writing "what a user of the code would wish there was". This is probably the best place, mindset and moment to define those specs in detail.
TDD encourages you to write many times the number of lines of code for your tests as the code you're testing, often 10X or more. So when you inevitably decide "there's a much better way to do this" (which happens to me 100% of the time) then you're not only changing your code, you're changing all of those tests. That's a lot of weight that you now have to deal with.
Most of the time, that's enough weight that the better design simply doesn't happen and becomes yet another chunk of tech debt that prevents certain things from happening in the future. I've seen that happen, and it's given me a very bad taste in my mouth for "TDD" because "TDD inertia" is the reason that better designs aren't implemented. If that piece of software lives for a while, and grows in scope, someone is going to have to deal with that, and eventually make the architectural change anyway.
By all means, test your code, of course. If you can find a way to easily generate test cases for an arbitrary code base, by all means, do that, because the test cases are no longer influencing your decision to change how your application works.
Otherwise, being "test-driven" is bad, IMHO. Software development is never as simple as the various dogmas would lead you to believe.
Isn't the point of TDD that you can change implementation at will, safe in the knowledge that the tests will help guarantee that you're not inadvertently changing behavior?
If you do not have TDD, I guess the reasoning is that either you do not have tests for this (probably bad) and if you do, less of them (and so more easily changed).
Not sure it's an argument against TDD, but I guess if management/the programmer do not know about sunk cost fallacy, it might make them hold on to bad abstractions/layers.
Most of the time you have classes that interact with one another. If you test each class individually, and you decide to refactor what classes do what, you'll need to rewrite a lot of tests, even though the higher level behavior should not change.
Usually however, what I see (in Java and Spring Boot land) is test methods which use mocks that are set up to expect certain method calls, e.g. when(accountService.getAccount(id)).thenReturn(account). Typically there are anywhere from six to 20 lines of this stuff per test (and, it's rarely de-duplicated into a separate method). So as soon as the contract to getAccount() changes, a bunch of tests need to change.
Second, it's common to believe that the "unit" in a "unit test" is either a class or (less often) a method. Consequently, people write test classes for every single class of a group of classes that are acting in collaboration, when what they really ought to do is write tests against the single class supplying the public interface, varying input to that class as necessary to achieve good confidence.
See specifically the "Fig. 5 - Test dependencies" section of the post.
Personally I think mocks are far over-used in tests, and I much prefer the solution kovarex outlines. I think dealing with mocks are the bigger source of test-update friction here than the tests themselves. I was already layering my tests instead of using mocks. As in having no clear line between "unit" and "integration" tests, everything just uses the "real" implementations of things and tests just get naturally more complex the higher up the stack it's testing. The idea of sorting by dependency depth is a cool idea I hadn't ever considered, though, I'll be borrowing that idea.
If that happens I'd say you're doing TDD too early. TDD can be very useful, but early on in the process of transferring a design to code, all your interfaces are highly unstable and there's a lot of experimentation, so sticking strictly to TDD would naturally be frustrating. I would start a bit later when at least some of the API has stabilized and you don't have to do major changes.
TDD helps with creating user friendly APIs, since you experience it from the user's perspective. And it forces you to actually write testable code and not incur technical debt that's very costly to remove later (having to refactor code to make it testable).
> Most of the time, that's enough weight that the better design simply doesn't happen and becomes yet another chunk of tech debt that prevents certain things from happening in the future.
This reads like you're saying that tests themselves are technical debt...? Because you're going to run into this issue (or should run into it) regardless if you use TDD or not. Eventually you'll want to refactor parts of your codebase and, sure, you'll have to change some of your, hopefully mostly, unit tests. So in that sense you can say that doing TDD early creates a lot of technical debt that needs to be resolved quickly, but like I said above, that doesn't have to be the case.
TDD can be a dogma just like any practice (Agile is my favorite), but it doesn't mean that it's not useful if used correctly.
Kudos to the Factorio team for adopting it, which I think is rare in the gaming industry. The idea alone of testing the complexities of a video game is mind boggling to me as a web developer. Especially in an industry where it's popular to hype and sell broken products with the promise of patches and DLC.
yeah TDD is a mess
Writing and maintaining a test before any experiments have been done with the design would indeed be frustrating and lead to a lot of rewriting. If you first explore the design and allow it to settle into a usable interface and then write the test for it as assurance that it's stable, you'd have the benefit of being able to safely refactor the feature and quickly add more tests for it. As long as the deliverable includes tests with decent coverage, who cares whether you wrote a test first without writing a single line of the implementation?
you haven't been talking to the same people I've been talking to, I guess. TDD is sacred and must be strictly adhered to, according to them. "anything less is just plain testing and is useless."
and people sometimes ask why I have such strong opinions against things like this. it's because people who don't write software come up with these things, and their lack of experience hides the realities of software development from them. they mean well, and they cause harm.
The better way to test your specification and behavior, IMO, is to use comprehensive E2E tests. The traditional test pyramid is upside down. If you have an E2E test for every user story in your spec, you can develop with confidence that your new code will not disrupt your users' activity. Unit tests are cattle, E2E tests are pets.
--
Dealer: "Hey? You there? I got some meth for you."
Me: "Go away! I don't want any!"
Dealer: "Oh now don't say that. You remember how good it is? I know you do."
Me: "I can't, I can't afford it, the price is too high."
Dealer: "What? Come on, it's free! You already paid for it!.
Me: "I'll lose my job, I can't, just go away!"
Dealer: "Your JOB? This IS your job. Open the damn door, THE FACTORY MUST GROW"
Me: ::cowers in the closet chanting please leave please leave please leave::
--
Also it was never good. It was more like a mind virus. Like the sort of problem or project at work you can't stop thinking about until it's done. Only with Factorio, it's never done. Never.
My best defense against it are other video games I can stop & start when needed. Or booting up my VPN connection and picking a work task from my back log until the cravings go away.
> (2) This allows you to instantly test what you write, and mainly use tests as specification
> (3) the problem comes when you break something and a lot of tests start to fail suddenly
My favorites. Don't expect to give away entire quarter, but at-least sometime would definitely be nice. All three so fundamental, and often ignored. In my experience, you get these right, it makes developer life so much easier. As someone earlier mentioned in thread, TDD is kind of like REPL driven development.
I think, one immediate benefit of companies focusing on good code is that engineers can aim for much more ambitious projects, and they can be more brave with the codebase. Instead we often end up with 100 over-engineered components with no well defined/enforced contracts, and a set of monolithic tests which runs the entire stack to test the most basic case.
The problem is, TDD is a poor development methodology.
I've found functional style of programming, or just decomposing functions into smaller functions, often work better with TDD. I'm personally not as much into pure TDD, as much as writing code that can be tested quickly and easily. In the end, what I want is when I'm writing code, I should be able to quickly run that specific piece of code and verify the some basic scenarios, instead of waiting a minute to push to cluster, another couple minute starting whole process and then running actual test.
First you develop fast and breaking things with alpha / beta versions. Then you make it stable with bug fixes and minor enhancements. Finally enforce the code with review, unit tests and code coverage, etc.
Theoretically they already have a good game engine (long lasting product) and interested in developing it further. Without enforcement, any future changes have potential to break things. Unit tests (enforcement) reduce that risks and make any changes / refactoring closer with specification.
Maybe the devs haven't found a worthy new project yet?
Maybe they just want to leave their code in good shape so they (or someone else) can come back to it at a later time and pick it up relatively quickly.
And that's got a non-trivial chance of happening.
I am no right-winger (in fact, I am a socialist), but I understand why Kovarex said what he said, and I think he had a right to do it. Yes, civilized discourse is important, but it's also important to respect individual freedom not to be pushed into such discourse from random Reddit hecklers.
I suppose it offends your sensibilities not because Kovarex did anything immoral, but because of his word choice. And honestly, I hate this surface level of analysis, where we judge the morality of a person by looking only at the words they use. It's counterproductive to addressing real problems, as there are far too many real villains in this world that are very nice in person.
You're free to read my comment history on this subject matter rather than just making an assumption and patronizing me. You'll probably find that it's more persuasive to actually engage with people than treat them like an idiot right out of the gate.
> but I understand why Kovarex said what he said, and I think he had a right to do it
Who claimed that he didn't have the right to say that? This is a weak straw man; nobody is claiming that he can't say that, merely that he shouldn't.
> I suppose it offends your sensibilities not because Kovarex did anything immoral, but because of his word choice.
... yeah? That's kind of how people work. How you communicate matters. Pretending that we should all just ignore him telling a fan to "shove it up your ass" is to basically ignore how humans actually work and think.
> And honestly, I hate this surface level of analysis, where we judge the morality of a person by looking only at the words they use.
As compared to what, mind reading? Literally all I have is his words online, and they're toxic and combative.
I also really dislike that you're upgrading this to "morality" when my original point was about whether I wanted to do business with him. I feel like this is a neat way to side step my freedom to not do business with him.
> It's counterproductive to addressing real problems.
So what, I can't make my own decisions based on someone else's behavior because there are other issues? I'm obligated to give him money after he behaved like an asshole because reasons? How absurd.
https://www.reddit.com/r/factorio/comments/o2ly6f/friday_fac...
If that’s not enough for you, this gem seemed particularly objectionable to me: https://www.reddit.com/r/factorio/comments/o2ly6f/comment/h2...
No, there is no practical difference between the "can't" and "shouldn't", if your intent is to participate in the punishment for what someone does. (Maybe you feel, as an individual, you cannot influence it. But in fact you can easily become part of the mob that does have the real power of punishing somebody.)
> Pretending that we should all just ignore him telling a fan to "shove it up your ass"
Yes, of course we should ignore it, it was just a heckler. Even a heckler can be a fan. Everybody can be an a* sometimes, and deserves to be called out about it.
> As compared to what, mind reading?
Compared to his actions, of course. If you don't have additional information, only words, think twice about acting upon such information.
> you're upgrading this to "morality" when my original point was about whether I wanted to do business with him
It's of course up to you. But why wouldn't you want to do business with him anymore for other reason than to uphold a moral standard? It isn't like he cheated you, in fact we probably agree that Factorio is a great product. The morality comes into it from reasoning about your response.
In any case, I was speaking in a larger context. People do this all the time. For example, there are people who think Linus Torvalds is worse than Bill Gates because of the harsh words he sometimes uses (in public).
To me, that is a very superficial way of looking at things. And same here, I think Kovarex is one of very few capitalists who actually deserve all the money they got.
And that's so sad about all this "cancel culture" thing. It's boycotting people who are not necessarily the worst, just easy targets.
> So what, I can't make my own decisions based on someone else's behavior because there are other issues?
You can do whatever you want, all I am saying is you "shouldn't".
Oh the irony of attacking free speech in order to protect it. How, prey tell, do you intend to differentiate between those who are using their speech legitimately and who is part of "the mob"?
I don't think I need to do it (why would I have to?), but it seems pretty straightforward. The online lynching mobs cross the line when they cause real world consequences for the lynched. (However, it is hard to predict in advance that the mob will happen and what the consequences will be. That's why people should be generally cautious about such participations.)
In other words, the punishment should be proportional to the crime. Kovarex got his comment deleted and that should close the issue and end the drama.
I think Ken White covered this dichotomy best, in this case a few years ago over a guy name Pax who got fired for his intemperate speech. See below.
> The foundation of "witch hunt" rhetoric is the notion that some free speech (say, Pax's) is acceptable, and other free speech (say, the speech of people criticizing and ridiculing Pax and his employer) is not. You can try to find a coherent or principled way to reconcile that, but you will fail.
And I also think that the whole point of free speech is to be free of societal consequences of it, whether they have government authority stamp or not, because it's IMHO impossible to discern that either (and I think the former Soviet regimes, which started in good faith of inclusive community, and ended up authoritarian, show exactly that problem).
What an extremely weird take. Are you saying I don’t have the right to decide who to do business with based on what they say, or that that’s somehow inappropriate? Restraining the right of free association to protect free speech is an awful idea.
> And I also think that the whole point of free speech is to be free of societal consequences of it
You are wrong. Full stop.
Free speech is about the right to say what you want without the government punishing you for it. This also includes the freedom to express disapproval or opprobrium. The ability to express disapproval is fundamental to free speech, any attempt to curtail that is in fact an attempt to curtail free speech itself.
If you wish to speak without social consequences, speak anonymously. This is the exact reason why the Supreme Court has held that anonymous speech is a right, to speak controversial ideas without suffering social consequences.
Also, you’re criticizing me for considering not buying a future DLC. Under your own theory, aren’t you being anti free speech for criticizing me?
> whether they have government authority stamp or not,
If you can’t tell the difference between government action and individual people expressing an opinion, that seems like a problem specific to you and not this society.
> and I think the former Soviet regimes, which started in good faith of inclusive community, and ended up authoritarian, show exactly that problem
Ironic, given that you’re proposing authoritarian responses to protect free speech.
These arguments really just boil down to people disagreeing about each other's reactions to things. What somehow gets obscured by oblique discussions of ethics and politics every time I see this kind of exchange online, is that it's simply two sides disagreeing about what a reasonable response is. There's nothing really to discuss about that. I (and the other guy, probably) think your decision to not buy the game because of the developer's comment is a dumb decision. You think otherwise. There's nothing to argue about here.
If you're building a new game and your current GUI paradigm sucks, overhauling it first makes a lot of sense.
> Imagine you have a company that goes slower and slower every quarter, and then you confront the shareholders with the statement, that the way to solve it, is to do absolutely no new features for a quarter or two, refactor the code, learn new methodologies etc. I doubt that the shareholders would allow that. Luckily, we don't have any shareholders, and we understand the vital importance of this investment in the long run. Not only in the project, but also in our skill and knowledge, so we do better next time.
This isn't necessarily the full explanation, but it's certainly something to keep in mind.
> now there are 9 programmers
Companies on the stock market don't have "9 programmers". They have a lot of teams of 9 programmers. So while it's true that it would probably completely be impossible for a stock market company to completely freeze for a quarter or two, individual teams can still do that.
If the factorio team grows to tens of programmers (it probably won't and probably shouldn't), I would be very surprised if they find the need - and if they manage to - freeze all teams together for a big refactoring round. I am also unsure that it would be the right approach. That observation holds wether they go public or stay private.
I feel like testing your behaviors is pretty hard, and even if you unit-test your behaviors, there’s still integration tests.
I only write games as a hobby and never use TDD, even if I’d like to, since the tooling is just either poorly documented or too slow, or both.
Usually this ends with me being frustrated with the slow development cycle and pushes me towards more unconventional methods of developing games in Javascript using Mocha to run the tests directly in the browser.
Modern games are notorious for using the first, most loyal customers as beta testers (hello Fallout 76...).
The reason is two-fold... while you absolutely can test some parts of the engine (e.g. collision detection, networking) you can't really "test" stuff that needs a human eye to see if it's working as intended (anything that's rendered) or involves randomness (e.g. fire, fog, water, opponent spawning, loot). That means you have to hire lots of skilled (!) humans, provide them with expensive rigs, and give them time. Which is incredibly expensive.
Game QA is more involved than that, of course. Content needs to go through a signoff and QA process that involves humans (we do that, too).
I interviewed at a game company a few years ago where one of my daily tasks would be to spend an hour just playing the game and seeing if I spot any bugs. I didn't end up taking the job so I didn't see how involved their actual code testing process was, but it was apparent that they actually cared a bit about quality control.
Artifact did it too (which is no longer being worked on though).
Yeah it was a painful experience, but I survived.
I’ve seen engines which used them, but not games. The rationale was that it was just too hard, which always felt like a cop-out to me.
I would dearly love to have more automated tests in the game I’m working on now, but I’ve never seen a model of it working well which I could copy, and part of me suspects that it’d be a huge investment of time to figure it out entirely on my own when I’m already vastly overworked as it is.
If anybody has references to indepth case studies of making game engines more friendly toward automated tests, I’d be super interested to see whether there were lessons I could apply toward my own situation!
Continuous integration and testing pipelines in games - case studies of The Talos Principle and Serious Sam https://www.youtube.com/watch?v=YGIvWT-NBHk
Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours.
They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.
https://technology.riotgames.com/news/automated-testing-leag...
"Early Access" seems to be the more popular route now. People are happy to pay to test a game and feel like they participated in its development. Check out Star Citizen if you want to be amazed and depressed by this truth at the same time.
Sometimes it works well. Sometimes it doesn't. Seems largely dependent on how engaged the devs are with their community.
A simple example might be simply starting and closing the engine after the automated build & package process to make sure the game actually runs, but I've seen things like using bots to emulate player behaviour to smoke test gameplay functionality. No Man's Sky used automated tools to evaluate the procedural-generation algorithms[1]. Here's a more comprehensive example of automated gameplay testing[2].
[1] https://youtu.be/sCRzxEEcO2Y?t=3100 [2] https://www.youtube.com/watch?v=VVq_hgaX8MQ
One of my stumbling blocks at the moment is that the machines in my CI setup don’t have graphics cards in them (or monitors attached or etc); they actually can’t run the game, and I’d have trouble detecting the difference between “runs successfully” and “runs, but only displays a black screen” or other critical graphical issues.
My team writes distributed systems, which drastically reduces the value of a TDD approach. There’s only so far you can take the technique with a database backed api before it just becomes absurd.
You need a large QA team (relative to your team size) to test for fun anyway. The game is constantly getting tested and bugs will get logged. The marginal benefit of automated tests is less than other places because of this.
Games have no specification. You have almost no idea of even the genre of game you'll end up with at the end of the dev cycle unless you're making a sequel that has to fit into a mold. Sure you can write tests as you go along and test that enemies with negative health die. The next day someone will suggest "what if they stay alive for a period of time and then explode!" The definition of correct is constantly changing.
Tests ossify functionality. It makes it harder to change things because at the very least you need to also change the test. If you're just changing tests whenever you want to suit your new desires then its hard to build trust in the tests.
Games don't need to be correct. They just need to be fun. This also decreases the marginal benefit of tests compared to other industries.
That said, it would be natural to unit test some data structure or some well defined system. Also, once your game is done, a la Factorio, you can go back and write tests for some refactor because you know the full design specs.
To be fair, I don't pretend to know how CDProject developed their games. They might already have testing and the timetable was the problem.
It's interesting to think about the other HN thread discussion about comments vs one-time-call function abstractions in this light: https://news.ycombinator.com/item?id=27546135
I'm a big fan of "put code in one place" too. It was the biggest factor that convinced me that JSX was a great idea compared to separating the templating logic out.
IME most developer do not understand each test has four possible outcomes:
* Code is good and test passes
* Code is bad and test fails
These are the only two possible outcomes developers focus on: When I ask what they should do if a test that used to pass now fails, they always tell me stories about how to debug the code under test.
There are two additional possibilities in test:
* Code is bad yet test passes (false negative)
* Code is good yet test fails (false positive)
Again, IME, most people do not look at the test again once it passes for the first time.
As a result, tests which are themselves code, become the largest untested part of the code base. You get these thousands and thousands of lines of untested code yet you have 100% code coverage.
Some of my blog posts on testing:
* Deception in tests considered harmful https://www.nu42.com/2017/02/deception-in-tests-harmful.html
* Know what you are testing: The case of the test for median in Boost.Accumulators C++ Library <https://www.nu42.com/2016/12/cpp-boost-median-test.html>
* Who is testing the tests? https://www.nu42.com/2015/05/who-is-testing-the-tests.html
* Slashing one's feet with tests, or, how to fix 2,950 test failures in one fell swoop https://www.nu42.com/2015/08/fix-2950-test-failures.html
It has certain characteristics that align with a game like Factorio or Oxygen Not Included etc. such as visual programming, backpressure, common interfaces, local retention etc.
I can imagine this being applied to distributed/cloud computing as a way to reason about high level interactions and perhaps functional/integrated testing.
Possibly the google analytics is doing something heavy (although it doesn't look to be when spot checking with a profiler), but there's otherwise nothing JS that runs continuously.
I think Factorio itself actually runs better on this laptop than that Factorio blog post does.
It is called restructuring and big companies do it all the time. Investors allow it, though they are rightly suspicious - sometimes it is good, but often it is change for the sake of change and not change for better.
if your game has a lot of interactions, and you want to make sure that your changes are not causing unintended interactions, tests like these would help a lot during development.
But anyway, I doubt tests help at all in the prototype phase (by any procedure you want to get them). My guess is that they are incredibly harmful.
If only game development worked this way!
I also misread TDD as TTD (related to the trains in factorio) first