Yes, this is a good idea. before trying to make dramatic changes to internals (refactoring etc) it is a good idea to get some kind of automated tests around the system.
you'll get quite a lot of value from a small number of basic smoke tests / end-to-end integration tests.
> Giving contributors a way to run tests locally against their changes on their dev boxes, with a testing framework, would be almost as good as running the app in production
a nice way i've seen this done is to focus on automating a build script for launching the app and any dependencies it needs, running automated tests against it, then tearing everything down.
figure out how to automate into a script that developers can use to run automated tests
figure out how to automate the environmental set up (e.g. installing package dependencies)
once you know how to do those things reasonably reliably for dev machines, set up a jenkins CI build to do the same thing: check out the code, set up dependencies, then run the automated test script
Since you've got a web app, it may already be difficult to get automated reliable set up and tear down of dependencies under control. E.g. you use a DB - can you automate setting that up and populating it with test data before a test suite runs? Your app automates infrastructure tasks -- how can you test that without actually triggering the changes, etc?