Adopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/