Does anyone have experience with distributed locks and would be kind enough to share?
So, rather than lock large resources, we would break the processing into tiny tasks, each completely contained in a message object. Each message specified a new message to launch on completion. This way the messaging protocol is preventing resource contention rather than a lock.
[edit] The context involved some licenses software to which we only owned several instances. The original approach would attempt to grab a lock and run the batch. The final approach just created a queue for jobs and one consumer for each available license. Now, none of our compute is spent waiting on locks either.
If the data size is limiting though, you can embed a Redis key or Hadoop FS reference into the message, rather than breaking your message system with massive data.
Then, you can have the storage system listen for completion messages, and cleanup appropriately. Or, like we did, just shove out old data LRU style. This way some history is always available for debugging.