Comment by patio11 | Hacker News Reader

Comment by patio11 | Hacker News Reader

Parent Full thread

Autotest has been the single biggest instigator in me adopting a more rigorous testing procedure for AR than I have previously used. The number of bugs it catches practically before I leave the method I'm tweaking is astounding.

(I mostly do unit testing, but also do some testing against APIs to ensure that I'm passing them data in the format they expect. One of my external service providers had downtime yesterday, and autotest caught it about six hours before they did. I wasn't successful in waking them up, but at least I was able to push a hotfix to AR to minimize inconvenience while I waited for their API to come back.)

[Edit: I might consider, if tests were taking that long, spinning up a fleet of VPSes to munch through them for me, maybe with a continuous integration server. Unit tests should be disgustingly parallel, since they're small units of work and shouldn't depend on each other. That means if one Macbook can execute the suite in 15 minutes then you should be able to have 30 Macbook-equivalents running either in the server room or the cloud munch through the same test suite in 30 seconds, give or take a little startup time.)

We do have a CI server, and as you said it works well for catching failing tests. However, it requires that you commit and push your changes in order to test them, which means you are effectively publishing untested changes to your entire team. The same for any kind of distributed testing, unless you are using a shared volume to host your sandbox.

I'm running a Mac Pro with 8 cores, so there is a fair bit of parallelization I can do locally too. Unfortunately, the tests all depend on the database, and while I can certainly use tools like deep-test to spin up separate DB's for each worker, I've found that doing so adds a full 60 seconds to the test run. I fear that until we eliminate the database from (most of) our tests, super-fast runs will continue to elude us.

CI and distributed tests are good things, no question, but I'm still looking for ways to make it possible to run my tests locally in TDD-fashion. I'm far from out of ideas, it's just a matter of making time to experiment.

This is totally a personal/team comfort question, but is there any reason why you can't have two remotes? "git push jamistest" might use a few bits on a spinning platter somewhere, but that is cheap, and there is no reason your team has to see it if you don't push it to the master repo, any more than they see changes you keep on your local repo.

Aside from me simply wanting to be able to quickly run my tests locally, you mean? :) Mostly it's just an issue of configuring that so it works for all the programmers. Each would need their own remote, and each would need to be hooked into CI. Definitely possible, it just hasn't been a priority.

Most of the time you need only some of your tests (the ones you're working on currently). Then, preloading the framework becomes the real bottleneck. But it's solvable too, via spork (I believe, there is spork-testunit, but I never tried it).

Most of my coding is done in minute loops (add test / watch tests fail / add couple of lines / watch tests succeed). YMMV.

spork is friggen awesome, and if you are on MRI/Yarv you really should use it.

http://spork.rubyforge.org/

Hey Jamis,

Your CI server should be able to accept a patchset and run it without committing it. TeamCity does this (or roll your own CI server can do this too!).

There's actually a gem for your edit: https://github.com/jasonm/parallel_specs

Then there's hydra: https://github.com/ngauthier/hydra

And the excellent presentation on speeding up your tests with hydra that I can't find. It was one of those showoff presentations on GitHub somewhere...

I believe you're looking for the presentation "Grease Your Suite" hosted here on Heroku: http://grease-your-suite.heroku.com/

Absolutely! Thank you!

Video: http://vimeo.com/12705404

I cannot recommend this video enough.

It is a step by step guide with code showing how the guy reduced his suite's running time from half an hour to 15 seconds.

A lot of it is preventing so many db objects like in the OP, but it also discusses parallelizing the tests using hydra to make use of multicore. This ties into the db access thing as well though as to do parallell tests you have to make all the cases set up and tear down their own fixtures transactionally. Not doing this is a leading cause of slow tests anyway.

He even has metrics for stuff like running the whole suite off a ramdisk, the effects of SSDs, etc etc. He worked on this at his company for weeks/months apparently.

It's too hard to explain here, just watch that video if you do ANY ruby testing!

Reply on news.ycombinator.com