What happens when you scale up your dynos and you end up with multiple servers all coming from different IP addresses?
Heroku logs are labeled with the ID of the dyno.
I know the heroku logs are labeled with the process (web.1, web.2, etc) but if you authorize a set of IPs to splunk and then add 10 more instances with heroku, I would imagine splunk would reject the new IPs. I'm curious if the heroku log drain always comes from the same set of IPs regardless of the dyno.
I've been exploring different logging options recently, papertrail, loggly, logevents and now splunk. Papertrail was definitely winning for simplicity but I felt the first three really fell short when trying to filter the returned information. I'm using scrolls to generate the logs so they're in key=value format. I've just tried splunk and it blows the others away for analysis. The big difference is in the way you can filter the information, it understands the key=value format and lets you search and then filter based on those. With the charting and dashboards I can build views that really allow me to see what's going on at a glance.
I actually prefer PaperTrail for most tasks - the main difference being that you start off with the log and can scroll through easily (instead of having to fiddle with filters).