For us, a task is a predefined set of processes. An instance of a task is a stream, an instance of a process is a process instance. Typically, a stream will have a few process instances that are typically batch jobs sent to the batch farm. We also allow jython scriptlets as process instances to do house keeping, namely to instantiate multiple sub streams and define variables and the environment that might be required for a job. The root level task can have a version and a revision number.
For instance, a top level stream will have a process instance that instantiates 1000 substreams, which have batch jobs that can be sent out to the batch farm. Those complete, and the substream can send do more or exit based on the status and preconditions. The top level stream will end in success, failure, termination, or cancelation depending on conditions. Any stream or process instance can be rolled back, but only when the parent streams are all in a final state because typically the preconditions (onSuccess, onFailure, onDone, etc...) applies to all streams and process instances to avoid race conditions and data corruption.
We use Oracle to do all our bookkeeping. Jython scriptlets are ran on the main server (typically only used to control flow). We use email for batch system messaging. Typically our jobs are very long jobs and take multiple hours complete, especially monte carlo jobs, and because of this, email has scaled fine, because the amount of emails rarely exceeds 100/second, even if we are concurrently running 2000+ batch jobs. I'd like to move to 0mq for the slight benefit in performance, but really email has been fine.
Most messages are logged directly to Oracle. Batch instances will typically have a log file, but that usually ends up on an nfs server or somewhere a user defines. Previous executions are archived. A user will define a log root, and the log file would typically be in the form of something like:
$LOGROOT/TaskName/TaskVersion.TaskRevision/StreamID/StreamID/ProcessInstanceID/log.txt $LOGROOT/TaskName/TaskVersion.TaskRevision/StreamID/StreamID/ProcessInstanceID/archive/executionID/log.txt $LOGROOT/TaskName/TaskVersion.TaskRevision/StreamID/StreamID/archive/executionID/ProcessInstanceID/executionID/log.txt
This can sometimes lead to directories with a very large amount of files,
We abstract out the batch farm and have custom daemons so we can connect dispatch jobs to various batch farms around the world.
So far, it's scaled pretty good. We have over 30 million streams and over 60 million process instances logged in our database. Some of the tasks are pretty complex tree structures do to the parent-child relationship and nesting we allow. This is one reason we've stuck with oracle, due to the fact that it has a CONNECT BY statement. In practice, that's been okay, and would continue to be okay if we had a good partition plan. We have plans to move away from Oracle and implement a database agnostic solution based on temporary tables. Due to legacy requirements and uptime necessity (this software has been developed for the last 5 years), it's been hard to implement these fixes in production. We plan to overhaul some of this when we get a new database server.
Storage is typically up to users writing a task, who often use nfs or, more typically, xrootd (http://xrootd.slac.stanford.edu/) depending on requirements. Many will register the data with other software I develop, called datacatalog, which is mainly for record keeping and metadata maintenance for the data. Access for this is typically done through jython scriptlets.