Rather than store passwords or sensitive config data in a file I'll usually try to store them in an environment variable. Heroku has a little guide on this: http://docs.heroku.com/config-vars
...this kind of thing doesn't have to be Heroku-specific, though.