We uploaded the generated files to S3 at the end of each individual job. We could have used AMQP's 'ack' for each message, but it turned out not to be necessary.