How we took our tests from 100 seconds to 100ms

> Use sqlite :memory: for tests

Then you run into issues where the functionality works on SQLite, but not on your real database.

Hi Shane, thanks for the tip about sails-memory. We've seen a lot of inconsistencies in the way Waterline behaves, so we're hesitant to run it against a different backend than the one we use in production. We'd also lose all of the uniqueness constraints that we use in Postgres.

In general we're trying to write and run more tests that only create and manipulate objects in memory, especially for testing business logic. Unfortunately we still have to have some level of integration with the database to test things at a higher system level.

Re: transactions, most of our database code uses the ORM, so we thought other test speedups would give a bigger benefit. We recently added an interface for getting/querying with a `pg` connection directly and we're using that to implement transactions in a few critical places. I'm planning to release the code soon - hopefully in a blog post in a few weeks.

Disagree about DB tests. If you want to test queries, then it's bad idea to mock DB. If you don't want to test queries, don't call your tests coverage full.

There should be some optimizer for npm modules. The current system is silly. Popcorn Time installs around 5000 files as part of its node_modules. How is that useful?

There should be some way to combine things into a single module/file or otherwise glob it all together, while removing pointless test, docs, and Adventure Time animated gifs from your deps folder.

Django also issues lots to stat()s when starting up their test framework.

Running against a local mysql instance running on a ramdisk and emptying the DB tables between tests yields "fast enough" performance while avoiding test pollution: ~300 full-stack scenarios in ~5sec start to finish.

In an alternate history, what would have been a good way for node.js to reduce the number of stat calls? What would the tradeoff be?

Should add, a problem with `rsync-auto` is some files in node_modules (bcrypt, among others) are compiled at runtime, and produce different outputs for Mac/Linux - you can't just copy them from Mac to Linux and expect it to work. Probably some things we could do to hack around this but it didn't seem worth it.

Other options for databases:

    - Run all your tests in a transaction, roll it back in the end (also avoids writes to disk in most scenarios)
    - Put the DB file system on a RAM disk

In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests.

End to end tests are great for catching connectivity bugs though.

Though you also have to ask how many additional bugs you'd have if there were no unit tests.

Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.

For me, the value of writing unit tests is in the writing of them more than the running of them.

That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.

Try to add fuzz-testing into your unit tests.

The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding.

You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway.

I think the nature of units tests make it easy to underappreciate them.

my experience is like yours that unit test developed in development are rarely useful.

however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.

Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour.

In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.

You can use both things...

It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.

> In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch.

Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code.

Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together.

This is only natural and does is no way suggest unit-testing was a wasted effort.

> Tests are not supposed to be written with speed in mind

This is why the tests took 100 seconds in the first place. While I agree that speed isn't the number one priority for everything, Reducing the length of time your entire test suite takes to run to be doable on every single run of your framework is absolutely worthwhile.

> Tests are not supposed to be written with speed in mind.

There's always extremes.

If people write tests with no regard for speed (as I've seen) you can have "unit-tests" run for 45 minutes before completing.

That might be OK to guard against regressions on a nightly-build, but it will in no way be a test-suite I'm willing to running prior to every commit I ship.

If the test-suite runs in less than a minute, I'm much more willing to do so before submitting code, and in that way avoiding regressions which needs to be fixed later on.

If the tests runs in less than 10 seconds, I'm probably going to run them for every major change I do, even if I'm not planning on submitting anything yet, just for peace of mind.

There's IDEs where tests can be run automatically in the background as you make changes and let you know the second you broke something. For that to work efficiently, your tests has to be sufficiently fast.

So maybe the aim shouldn't be that tests must be fast above everything else. But if the test-suite can be run quickly and casually, it's value will increase tremendously.

This is somewhat more reasonable. If your core business logic encompasses flushing data to lying disk controllers and being robust in the face of kill -9's, the root filesystem filling up, not corrupting the database on segfault, and not falling over on multi-terabyte datasets then your tests might take a long time.

I just about choked on my coffee. If you're serious, is the issue just the size of the test suite, or are the tests themselves just woefully inefficient? By the time you're done, you've forgotten whatever it was that you modified.

I worked on a perl application server for a big ecommerce company. We'd stat so many files. Pulling in the meta-object library ('use Moose') was guaranteed to make your program take a long time to start.

Any link to a discussion on these Ruby stat calls? We might look into them as well.

Well, to be fair, one of the patterns I've seen with Rails projects is to move as much out of Rails proper as possible, and treat it just as your web/(maybe)database interface.

Yep - we've run into a lot of really unexpected behavior with it and with the ORM (Waterline), so we try to restrict ourselves to "the minimum possible framework subset that gets the code to run".

For example, we disable `dynamicFinders`, don't use policies, don't use blueprints, don't use Waterline's associate/populate/destroy, don't use the "turn any controller function into a named route" setting, don't use globals, disabled all of the Grunt/websockets junk, and overwrote all of the error response handlers.

The result is that day to day we're writing something that resembles an Express app with its own `routes` file. We're looking to move off of Waterline/Sails entirely, it's slow going though since we're a small team and it's tough to prioritize that, as it "works" now.

Every company has parts of their tech stack that don't do a great job - we're somewhat fortunate this one is easy to mitigate.

Never fear, our new project compiles Sails.js apps-- the core of what we're trying to solve is exactly that (the transition between using a more heavy-handed framework, and being able to customize/optimize/understand minutia as needed). More on the tech here: https://www.youtube.com/watch?v=nZKG7hLhbRs

Being able to move "off" of Sails, interestingly, is a design principle of Sails itself. Sails is built on common node.js modules (express, socket.io, etc) and uses standard express middleware/APIs so that pieces of a Sails app can be ineroperable with other frameworks and tools. Waterline (the ORM layer) is designed to work with any database so that you aren't locked to a single vendor. It'd be hypocritical if we wanted to liberate you from other tools, but try to lock you into using Sails, wouldn't it? We feel that that's a feature, not a bug.

There are much larger companies (Fortune 100) that use Sails.js. This isn't a knock against Shyp at all, and I'm glad they're using Sails, I'd just like to clear up whatever the source of your misconception is. Sails, Waterline, and the many related modules are all in active development.

(Disclaimer: I work for Balderdash, the company that created Sails)

Sorry to hear that Andres - what version of Chrome are you using on Android? Will try and see if we can figure out what is going wrong

At my startup (35 devs on the main product), we'd spin off a dozen AWS instances so the tests would run in 20 minutes in parallel. And only tested commits could be merged into master. We didn't feel it as a bad situation.

There's also this: https://www.npmjs.com/package/require-times

If your suite run in 100ms, why wouldn't you?

That is probably on a CI server. During development, I'd probably be running way more than that, at every production or test line written, having a task that watches for file changes and re-runs all tests. Then, having a fast test suite is key.

While speeding up the process is always good, tools like lucifer or spork are only masking the problem: loading the whole environment for unit-level tests. Sails, like Rails, promote tight coupling between all parts of the system, which harms test speed, thus harming another development tool: TDD.

The more decoupled and isolated you write your business logic, the easier and faster it is to test drive it. Then you simply plug an HTTP interface (with routes and whatnot) that calls into your core.

hybrid_material - here: https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9... and here https://bitbucket.org/kevinburke/small-dotfiles/src/e4e1baf9...