TTNT: Test This, Not That
github.com
github.com
Decoupling from your persistence (and presentation, fwiw) brings a lot of benefits, not only faster tests. I'm sick of people saying "why introduce a layer here if we'll never change the database?". The answer is simple: yes, you will change the database, specially when testing where you want to plug an in-memory store.
Again, I get that Rails gives us conventions so we don't keep bikeshedding, but how often do we challenge these conventions? How often do we question whether the problem we're solving fits the CRUD/MVC patterns?
Most of the Rails apps I've code reviewed have lots of tests of business logic that are tightly coupled to ActiveRecord and are (as a result) very slow.
One way to improve upon this might be to move from a test-oriented approach to a contracts approach where domain objects obey contracts.
It seems that this is the actual intuition behind the "declarative" associations in ActiveRecord, yet so many tests simply test what has been declared, as if it can't entirely be believed.... Does this model actually have a working input validation? Does it have working associations? If this is in doubt there are bigger problems.
Also, most apps use relational, hierarchical data (which is actually just a graph), which we declare via AR associations... models contain a mix of logic and data plumbing, and thus persistence layer concerns tremendously complicate testing simple graph oriented intuitions about our data, which could also be stated declaratively and enforced contractually instead of via the typical "callback hell" approach. So we stub callbacks and mock adjacent graph nodes and tell ourselves falsely that our "model" is well tested.
The complexity in testing most apps lies in setting up app state to test edge cases in the algorithms. There have been lots of workarounds to try to skip this or make it faster, but in the end the abstraction is the problem.
I want to be able to create simple data, such as a few hash literals, and test the flow of data through the business logic (algorithms) of my code without having to wait 60 seconds for the app to boot.
Ruby is not all that slow a language. I think a reasonable test suite for a large app should run in under 2-3 seconds. But Rails apps end up testing ActiveRecord, template rendering, and all sorts of other code that is already well tested and should be assumed to work by the test suite, but generally can't be because of the tight coupling mentioned above.
One other example: Templates are essentially functions that output HTML, but since they have ill-defined input requirements, there is not a simple way to throw valid input at them and verify correct output other than simulating the entire server request/response, yet most programmer errors will be in the implementation of the template, not in the boilerplate controller code. Yet millions of hours of CPU time (and programmer time) are burned testing these the hard way.
- Moving data into and out of the database, the "Data Access Layer"
- How the model objects interact and behave once moved into memory, the "Domain Model"; this is pure OO and is amenable to fast, in-memory-only testing.
Rails treats these two as a single layer. On a project I'm working on now, we separated them into two distinct layers, which allows 1-2ms test execution in the model layer. We do test persistence, and it takes some time, but we can be a lot sparser with these slow, heavy tests.
In the end I think the choice of approach depends on how complex the domain model is. My domain has almost 100 objects arranged in various ways, putting us over the threshold where persistence should be treated as its own concern (vs. bundling it into the model). I suspect a lot of apps are better done via the Rails approach as it's much simpler, if slower to test.
I have been inclined to agree with this, but lately have begun to wonder. I think the relational model (and document model) lead to the need for lots of manual traversal/management of the graph/hierarchy of a domain model.
And, in an OOP sense, models have behavior (and of course state).
But behavior involves mutating the state of one domain object and others, via their public API (behaviors). Yet most of the so-called behaviors are actually just adding/changing nodes/edges to the graph, which makes the model's main behavior data validation.
So we have this elaborate, tightly-coupled system that pretty much only does data validation, but via the elaborate mechanism of object behavior and internal state.
The typical test suite tests the propagation of this data through the system and verifies that the proper object state has been mutated/persisted, since the proper functioning of the system is easy to doubt.
I came from using Rails to doing DDD-style C#/.NET development. It feels very "enterprise" but then, there are some very good lessons I've learned, one being how to write fast code-only unit tests.
If using RSpec, there are "view" tests that test rendering the html in isolation. You specify the instance variables (AKA input) that the template is expecting, and you test the output. No server request/response is involved.
See https://www.relishapp.com/rspec/rspec-rails/v/2-0/docs/view-...
Honestly if you can't eliminate the persistence layer, I'd rather see DB vendors put more time and effort into making in memory tables work. The Java people (hibernate) swap out the store for an in-memory one to speed things up, but I'd rather see a Postgres instance with heavy write-back caching for testing.
I've since had projects with 9ms per test and it's hard to go back to thinking 80ms is 'good'.
> Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives)
The problem is you're going to need to run the entire test suite before deploying anyway unless the algorithm is perfected. If you have to do this anyway, while it's helpful to have more confidence going into an hour long test build that it's going to pass, it's probably more helpful to just build an infrastructure on which the entire test suite runs more quickly. I've worked on sites with extremely large test suites and it was never an issue to run the tests in < 10 minutes because they could be run on an infrastructure that allowed tests to be run in parallel. I think that's the solution rather than a complex and flakey algorithm.
I haven't read the code, but the associated document explaining the method makes no mention of accounting for line renumbering as a result of changes.
The accuracy with which changes are identified depends on the diff algorithm. I imagine it is more likely to generate false positives than false negatives (which is preferable), but I am sure there are cases where a line is unchanged textually but is different semantically (e.g. by the insertion or removal of an 'else' above it), and the diff program will consider it unchanged.
In my prototype this caused effectively any change outside of a function body to end up disabling the optimisation entirely :(
Tests are there to fight our natural tendency toward wishful thinking. Tests that fail for no reason enable people to keep thinking there was nothing wrong with their code. Meanwhile other people are twiddling waiting for them to fix something they don't think they broke.
Also, the build tool Buck (https://buckbuild.com/) enables this because it knows the graph and associates tests with the code they test...so you even get this test minimization locally as a developer (that is, outside CI) for free. I'm sure Bazel likely has a similar feature too.
Does anyone know if there is a similar project for Java?
IMHO, the build info at the top is pretty useful
And it's not just my one click -- it's the cumulative clicks of every visitor to the site. With it on the front page of HN, there are going to be staggering numbers of potential users glancing at the page on their phones during their commutes today, not seeing anything immediately eye-catching, and moving on to the next submission.