Cannot reproduce the exact same training results – diffusion models

Hi ! In order to collaborate with my team we want to be able to get the same training results based on a finetuned version of https://huggingface.co/runwayml/stable-diffusion-v1-5.

We have real issues defining the exact same environment, even if we have the same code/ use the same args with weight & biaises. We try to do our best with some versioning of the image datasets but it seems that the issue can also come from CUDA versions and the different libs (I also read it can come from the GPU being used).

If you have some tools that could help us replicable the same training environment that would be great. Not sure exactly what's wrong in our config.

Thanks!