I was thinking about similar concepts and problems then I decided to create a framework [1] that could possibly solve those problems.

I created a lightweight system, that serves web pages, added proper authentication, storage (mongodb), memory storage/cache (redis) , simple ORM (mongoengine), automatic backend, background processing/scheduling (celery), email integration, single command deployment (automation)

I have also demonstrated how to do scrapping/scheduling/email sending in this tutorial:

https://medium.com/project-enferno/create-your-first-minion-...

and this one explains how to automate the deployment of everything you create with the framework with a single command:

https://medium.com/project-enferno/tip-deploy-enferno-framew...

Hope someone will find this helpful.

[1]:My framework is called Enferno: http://enferno.io and runs on top of flask.