I'm going to try this after releasing Appointment Reminder.
For solo Rails developers who want the automation but don't want to deal with big icky configuration, can I recommend autotest?
http://ph7spot.com/musings/getting-started-with-autotest
I have mine configured to pop a little window on Ubuntu any time a change causes a test to fail. It is quite motivational and has saved my bacon many, many times these last few weeks.
It isn't really great for using for e.g. running a suite of expensive Selenium tests frequently, but it cranks out simple unit, functional, and API tests.