Slow database test fallacy
david.heinemeierhansson.com
david.heinemeierhansson.com
He opens with this:
> The classical definition of a unit test in TDD lore is one that doesn't touch the database. Or any other external interface, like the file system. The justification is largely one of speed. Connecting to external services like that would be too slow to get the feedback cycle you need.
No, "unit tests" in TDD -- and long before, TDD didn't change anything about the definition -- are tests that, to the extent practical, test all and only the functionality of the specific unit under test, hence the name. That's the reason why external interactions are minimized in proper unit tests (whether or not TDD is being practiced). TDD observes that such tests are generally fast, and builds the red-green-refactor cycle around that fact, but speed isn't the justification for the isolation, isolation of the functionality being tested from other functionality is the point of unit testing (which is designed not only to identify errors, but to pinpoint them.)
ActiveRecord at its core is meant to generate and run SQL and give you object graphs back, and you need to test that the SQL it generates is correct and does the right thing.
I think one of the fundamental, if not really directly addressed, divides between the sides here is over the question of whether domain models "using" (being tightly coupled to) the persistence layer is sensible architecture even outside of consequences for testing.
He also seems completely unaware that there is an entire school of TDD/BDD that doesn't really like mock objects (the Chicago vs London school http://programmers.stackexchange.com/questions/123627/what-a...).
I've just generally stopped listening to what he says on the topic. It doesn't seem to match the reality of what folk actually do.
Actually I think it's more his critics which misunderstand this—he argues against TDD, and he doesn't think unit testing is sufficient, but he doesn't argue against testing, or unit testing.
But what he describes as "TDD" to argue against it isn't TDD (either in his description of its substance or his description of its rationale), and what he describes as "unit testing" to argue that it isn't sufficient isn't unit testing (again, either in substance or rationale.)
> but he doesn't argue against testing, or unit testing.
He specifically argues that unit testing should be deemphasized if not outright eliminated, and that the reason for this is the elimination of test-first as a design practice. So, yes, he does argue against unit testing (of course, the argument is nonsense, since unit testing was an important practice before test-first practices, and test-first practices are independent of kind of testing -- sure, TDD emphasizes unit test first, but ATDD/BDD focus on acceptance test first; moving toward or away from test-first as a practice is completely orthogonal to the degree of focus on unit testing vs. other kinds of testing.)
You know what could increase the speed dramatically.... decoupling.
I also think decoupling phrased in the context of the Rails 2 to Rails 3 upgrade, where pretty much everything changed, makes perfect sense. Imagine just having a few wrapper classes that spoke to Rails and only having to adapt them. Sounds good to me!
Bernhardt: Boundaries http://www.confreaks.com/videos/1314-rubyconf2012-boundaries
Weirich: Decoupling from Rails http://www.youtube.com/watch?v=tg5RFeSfBM4
Wynne: Hexagonal Rails http://www.youtube.com/watch?v=CGN4RFkhH2M
Also, did you read the rest of the post? Typical test cycles are much faster than that. But a full test of the entire system takes some time. As it should.
This is also why the test suite for git takes a long time, but has a very good signal to noise ratio.
If only that "unnecessary code" had tests... Oh wait.
Not to mention you have increased the workload of everyone just to mitigate the impact of additional code. Let me know how that works out for you. :)
Seriously, straw men not withstanding, this thread is specifically about tested code where a complaint was raised about how long the tests take. And the "how long" was only 4 friggin minutes for a full suite, or 4 seconds for a module.
I don't really mind if a given branch is marked red on CircleCI. master must always be green, but a branch, not so much.
Say I change a model. I'll probably run the unit test for that model (4 seconds), then just push up the branch and forget about it. 9 times out of ten, I didn't break any other tests and everything's fine. Once in a while, I did, and 3-4 minutes later Circle via HipChat let's me know I need to check it again.
It's a bit meta, but this process is just like actual Rails apps—when the task can happen asynchronously in the background, it usually doesn't matter if it takes a couple minutes.
It differs very much across projects, full pouchdb test suite takes ~4 minutes, full firefox test suite takes hours, for rails ~4 minutes sounds entirely reasonable (kinda faster than I would expect) and wouldnt give me no reason to think about reducing the coverage and reliability of the tests by decoupling anything.
He mentions per model, which is what you're doing while doing dev, it's 4 seconds.
No.
I'm not interested in pop culture; I'm interested in being a better developer, and that requires a highly critical process of evaluating my practice. It's not enough if something works once, I want to know why it was effective there, and when I can use it. I want to try practices like TDD just to see how they affect the design, and then decide if I like that force. I'll use hexagonal architecture on side projects just to see how it helps, and if it's worthwhile. In short, I want to continue to study the art of software development rather than trusting emotion-laden blog posts with something as serious as my skill.
I don't believe Rails is so special it warrants revisiting all of the lessons from the past we've learned about modularity, small interfaces, and abstraction. It's just a framework.
Many codebases don't heavily decouple from their frameworks unless they have a good reason to do so, as they lose the productivity benefits of the framework in the process. The framework you choose to tightly couple against should be your flex point -- you don't have to design your own!
The level of tradeoff depends about the framework in question. I can recall a moderate-sized project where we bound against Hibernate ORM for a couple of years and eventually had to switch to MyBatis for a variety of reasons. But since we were using JPA annotations mostly, the coupling wasn't so tight to make the switch all that hard or brittle.
There are times where Hexagonal architecture makes total sense (immature frameworks, shifting dependencies, etc.) , and times where it doesn't at least for certain "ports" (you're building a moderately complex Rails/AR app, why bother isolating AR).
Uncle Bob is saying that Rails is not your application, your business objects that contain all your logic shouldn't inherit from ActiveRecord::Base because that ties you to a specific version of a specific framework (have fun migrating to a new version of Rails!) and means you have to design and migrate your schema before you can run any tests on your model code. You should be able to test your logic in isolation and then plug it into the framework.
DHH is saying that if you're writing a Rails application, of course Rails is your application. Why waste hours adding layers of indirection that make your code harder to understand, just to make your tests run faster?
Of course if it's just a prototype, who cares? But I really agree with Uncle Bob that tightly coupling your application logic to (a specific version of) Rails/ActiveRecord is a bad idea if you want to make a long-lasting, maintainable application of any non-trivial size.
Any time you introduce an abstraction layer to decouple some code, you're making a prediction. You're saying, "I think it is likely that I will need to change the code on one side of this interface and don't want to have to touch the other side."
This is exactly like financial speculation. It takes time, effort, and increases your code's complexity to add that abstraction. The idea is that that investment will pay off later if you do end up making significant changes there. If you don't, though, you end up in the whole.
From that perspective, trying to abstract your application away from your application framework seems like a wasted effort to me. It's very unlikely you'll be swapping out a different framework, and, even if you do, it's a virtual guarantee that will require massive work in your application too.
Sure, it's scary to be coupled to a third-party framework. But the reality is is that if you build your app on top of one, that's a fundamental property of your entire program and is very unlikely to change any time soon. Given that, you may as well just accept that and allow the coupling.
However, it's not reasonable to assume that your Rails 3 app will always be a Rails 3 app. You will eventually have to upgrade--if not immediately for feature reasons then eventually for security reasons. And upgrading a Rails 3 app to Rails 4 is a non-trivial effort, there are a lot of breaking changes, some of which affect the models (e.g. strong parameters, no more attr_accessible). If you skip versions you will just accumulate more and more technical debt.
I think that ideally, you would have your business logic in classes/modules that don't need to have code changes just because the app framework got a version bump.
But generally speaking you're right, the decision of whether or not to put in the up-front work to decouple your business logic from your application framework, is like an investment decision with costs and benefits. Uncle Bob is saying it's always worth it, DHH is saying it's never worth it, but I think the reality is that it's sometimes worth it, depending on you and your project.
There is a mindset out there that all coupling is bad. Uncle Bob's point of view that coupling to a specific framework is unwise, is one long held by OO design purists.
I'd rather prefer it thought that tight coupling is a tradeoff. You are trading productivity today for future migration risk.
If you know your app is a Rails app, and will always be a Rails app, then there's little reason to decouple. The question, is do you really know, or how reliable are your guesses?
Is it ?
I was under the impression that you don't include them because a unit test is testing a very specific piece of code and not the dependencies around it. This is why you'll mock disk/db/network, just like you'll mock other pieces of code.
TDD focuses on a particular method of leveraging unit testing to improve code quality and avoid bloat, but it doesn't suggest that integration testing should be abandoned. Integration testing is just outside of the scope of the part of the dev process TDD is focussed on improving.
It's also simpler to use unit tests for verifying all possible inputs and outputs of a component.
But, as others have said, unit tests alone are necessary but not sufficient.
It doesn't. TDD is largely about how to use unit tests to drive incremental development, but it certainly doesn't preclude any other form of testing being part of the lifecycle (it just doesn't have anything to say about them -- presuming that you have some integration and acceptance test practices, and that those are out of TDD's scope.)
With test-driven design (in an ideal world), if a change to an important module can break its own tests only, that means that it has a shallow and clear interface, decoupling its internals from the rest of the application.
Given that the module's code is exhaustively tested, that means there is a full specification of what it's supposed to do, given the various possible scenarios.
If the tests are well-written, that means the specification is comprehensible, which means the interfaces make sense in the domain's discourse.
Good TDD involves working hard to keep the code base clean and comprehensible. If you're only testing that everything works together, there's a chance that you're also less focused on maintaining good architecture.
I like testing those things on which my model depends. It gives me much more confidence. Why wouldn't I want to test them?
> as well as the correctness of the code that rolls back any side effects.
That's a drawback. No arguments from me on that one.
Those things all need to be tested, but if a single unit test fails, it's nice to know that it failed because the code was wrong, not because the database connection happened to die just then. If I have one test for the logic, and another that verifies that the database can be connected to, and a third that verifies the schema is right, then the specific combination of failing tests tells me a lot more about what's wrong and if my code even needs to be changed.
Much of the testing activity and literature of late has been complaining how brittle end-to-end tests are, because all the focus is on pure unit tests. This leads to defect pile-up at release time or at the end of an iteration. Whereas the smoother teams I've worked with did end-to-end and integration tests all the time. Unit tests existed too, but only when there was sufficiently complex logic or algorithms to warrant such a test, or if we used TDD to flesh out interfaces or interactions for a feature.
Many web applications don't have a lot of logic, they have a lot of database transactions with complex variations for updates or queries. So, especially if you have an ORM, which are notoriously fiddly ... it makes sense to have the majority of tests (TDD or not) hit the database, since the code will only ever be executed WITH a database.
Mocking or decoupling the database can introduce wasteful assumptions and complexities that aren't needed in your code base. The only time it makes sense to decouple the database is if expect you'll need polyglot persistence down the road and your chosen persistence framework won't help you.
I have worked with developers that prefer test cases run in under 1 second on every save. To me it helps to have a set of unit tests that are in-memory and very fast, that cover basic sanity checks like model integrity, input validation and any in-memory algorithms. But the bulk of tests really need to test your code as it will be used, which often involves database queries. At worse, use an in-memory database that can load test data and execute tests in a couple of seconds.
4 seconds is really slow, actually, and enough to take you out of flow. With a PORO Person object, decoupled from the system, that number will easily be sub 500 ms and possibly much less.
There's slightly different things that happen: now() will always return the same time, deferrable constraints/triggers are useless, you can't have another database connection looking at the test results or modifying the database (say you are testing deadlocks or concurrent updates, or you have code that opens a new database connection to write data to the database outside the current transaction), etc.
It's fine for simple, vanilla ActiveRecord use where you aren't using lots of database features, I suppose.
* at the start of a test run, create a new database, load the schema into it, load all the 'global' data that all the tests need into the database.
* write that data to a sql file using pg_dump --inserts -a
* before each test runs, I disable triggers (with set session_replication_role=replica), then delete all the data from each table, then load the data from the sql file back into the database.
This allows me to have data quickly cleaned out and restored on every test run and gives me real transactions during tests.
Okay... PostgreSQL is great but it still has a bit of catching up to do.
> ... run your MySQL
Wait, Oracle is an abomination but MySQL is okay?
> Before each test case, we do BEGIN TRANSACTION, and at the end of the case, we do ROLLBACK TRANSACTION. This is crazy fast, so there's no setup penalty.
You know what is just as easy? Making SQLite databases (aka files) for each test case. Copy a file, open it, delete it. It has the added benefit of allowing you to actually commit changes and not worry about rollback. There are some compatibility issues, and I'm not familiar with all those issues in a Rails context.
I think he was referring to administrative challenges and setup/startup overhead, not steady-state performance.
By doing this, you're breaking the dev-prod parity rule of 12factor apps [0]: you should make sure your dev and prod differ as little as possible. If you're using MySQL in production, you should also use it in dev.
First off, many people would argue unit testing should never really require a database. I'm not of that opinion, but I'm not writing CRUD apps, and since there's not really a SQL unit testing framework, using SQLite is a nice compromise, especially as unit testing should necessarily be limited in scope.
Integration testing should definitely be done on a system that is in parity with prod.
(Never do that on a production database, of course!)
With every test the questions should be answered are what bugs is this going to catch and which one will it miss, if you mock a dependency then you are introducing cases in which it will miss bugs and there should be a justification along with it.
This also means that a mistaken change to one important unit will not break the entire test suite. Sure, the entire program will break, but it's nice to get a single failing test.
Mocking also gives a very straightforward way to simulate interactions with collaborators. You just say "given that the HTTP module returned a 404, this request should be sent to the error log," instead of initializing those two modules and arranging for the desired case to happen.
There's a very old discussion about decreasing coupling and increasing cohesion that's super important to the whole motivation behind TDD and that nobody seems to be very interested in anymore...
Also, all that separation isn't free. Sure, I don't need to run all my unit tests every time I make a change - but if they're fast enough that I can, that's much less cognitive overhead than having to think about which tests are relevant and press the correct button.
innodb_flush_log_at_trx_commit = 0
http://interblah.net/the-problem-with-using-fixtures-in-rail...
Well, I've tried this before and it didn't work.
is this claiming 100% test coverage?