I would use SQLite between each of the stages so each process can be restarted, run concurrently and observed with queries.