I love Heroku for it's simplicity. They give you very few levers, making it less likely for you to screw up.
When we had a production add we faced following issues with Heroku:
1. Frequent downtimes. Heroku would have a 3-4 hour downtime every 2 months. In some cases, site would be down. In most cases you couldn't deploy and add machines.
2. There is an end user impact in terms of performance. After moving to Elastic Beanstalk, we had a 20% decrease in response time and 5% increase in revenue.
3. In the end Heroku is very simplistic. They don't have features like rolling deployment or Blue-green deployment. They had some beta version of rolling deployment, however that was buggy and caused version issues with deployment. With Elastic Beanstalk we have a health check based rolling deploy, which has caught production issues many times.
4. To do custom things, you need to deal with buildpacks etc, which is not very convenient.
5. Thought it wasn't an issue for us, Heroku can be expensive. Our server costs went down 80% while giving better performance.
So, in summary - Heroku is great to get you started quickly. However if you are doing anything serious but do not want to get deep into DevOps, consider something like Elastic Beanstalk.
Eventually we managed our environments using CloudFormation. That allows us to create repeatable Elastic Beanstalk instances. Once we got the CloudFormation template right, it was easier to maintain Elastic Beanstalk vs Heroku.