"...and hear your stories about how you’ve done this before."

Basically Mr Kurtz saying "The horror! The horror!" gives you the right impression.

For small datasets and short training times it isn't so bad, a deterministic training script on git and a dataset with gitlfs. So can easily reproduce the model when needed. Having to do this on very large datasets (that are possibly expensive to keep a copy of for just experiments and not prod) or really slow training times is basically disgusting. I'll be interested to see what you have done.