If anyone here wants to do this but don't want to implement all of this yourselves, this "field" is called Durable Execution. Frameworks such as Temporal, Restate and DBOS do a lot of the heavy lifting to get the idempotency, exactly once and recovery to a known state logic here.