The results of this paper would indicate doing what they did, but online could return better results
With our training recipe this can be easily done by accumulating the gradients across the entire batch and only doing one step with optimizer before sampling more responses.
In our experiments, however, we found the advantages of doing multiple gradient steps outweighed any potential drift in policy.
Ultimately the online-ness of data is on a spectrum and while more online data is better, other factors may be more important.
Bit pedantic, but amusing thought; wouldn't that imply that asynchronous actor critic is an offline training methodology?