If you have a similar use case, you should probably just setup a Kafka and Hadoop implementation. You could probably get a lot better performance by using a shared memory store to keep the intermittent states.