We used to use Scout. They're a great group of guys, but ultimately, we ended up just falling back to a Munin install. The biggest challenge for monitoring software seems to be the wide variety of environments you'll encounter on various servers. Scout does a nice job of focusing on Rails apps, but their scripts required just enough of my time tweaking that I ended up deciding that it wasn't worth the effort for the added insight we gained. Keep that in mind when developing your service. Make your documentation top notch, and provide an accessible support conduit for when users encounter issues.

Just to be clear, our decision to not use Scout wasn't because their service was bad. To the contrary, the service itself was very good. It just required a little bit too much setup and required a high degree of inference from the sysadmin. You more or less had to read the code in some of the plugins in order to make things work. That added up to more effort than I felt we were getting in benefit when compared to something like a local Munin install.

I'll definitely give you a try if you send me an invite.