Ask HN: Workflow for Scientific Computing in the Cloud?
Manually create amazon machine image (AMI) for experiment.
Manually issue command to start AMI on specified spot instance type.
Automatically connect EBS to instance for result storage.
Automatically run specified experiment, bonus if this can be parameterized.
Automatically terminate spot instance on job completion.
Something like docker that spun up on demand spot instances of a specified type for each run and terminated said instance at run completion would be absolutely perfect. I also know HTCondor can back onto EC2 spot instances but I haven't really been able to find any concise information on how to setup a personal cloud — I also think managing an HTCondor installation may negate the savings of all the automation. Do any other HN users have similar problems? How did you solve it? What is your workflow? Thanks!
NOTE: I posted this to Slashdot as well, but reposted here as I don't think it'll get much attention there.