I don't understand how my app could be set to idle when I have pings hitting it from three separate sources. Also, the pingdom ping is looking for a specific word on one of my pages. It has to find this word to be considered successful.
How often are the pings? Do the pings wait for startup or timeout? A lot of filtering methods on the traffic data could ignore isolated pings as well. They don't have to be using on/off. They could be using a higher threshold. E.g. one ping every 10 minutes, just startup and handle it then shutdown. 10 page loads every 10 minutes? Startup and stay up for 10 minutes. It's weird to even call page loads pings rather than the actual traditional ping as well, but whatever.
The pingdom ping is every minute (http check type with a string match optional setting). The heroku scheduler is running once every ten minutes. I'm not sure how often new relic runs.
Heroku scheduler runs in a separate dyno so that wouldn't impact idling of the web dyno.
In my humble opinion, there must be some other process going on in the internals of heroku to keep my app idled. Please explain it to me if I'm wrong.