This doesn't in any way require the training run or the build to be reproducible. It just requires the model, once released through the API, to remain available for a reasonable length of time (and not have the rug pulled with 3 days' notice).
We had 200 years of tinkering with relatively modern steam engine technology before Carnot and Watt started just barely scratching the surface of the first principles of thermodynamics and engine efficiency.
Even the eponymous Carnot cycle wasn’t rigorously defined mathematically during Carnot’s life. That being (T1−T2)/T1 as the temperature delta part of the equation, because absolute temperature hadn’t been accepted and defined by Lord Kelvin yet.
Some decade later the first law of thermodynamics was finally invented.
Hundreds of years of experimentation until the first principles. Machine learning has lots of control systems theory and information theory to help with analysis but we barely have an “engineering” in “software engineering” today, let alone in “machine learning engineering”. We’ll get there, but it’ll be awhile before there are proven design equations with rigorous derivations from first principle that allow us to design and build a precise AI model as surely as we can design and build a precise bridge or levee or distillation column.
Let the hacking continue, let’s not worry too much about the future “engineering” that will follow in its own time. Unless you want to discover it yourself or fund its discovery.
Maybe it's easy enough for them to just copy the model, tweak, hack and play with it from there with little interruption. No one really knows at the moment.
I meant to say that now the model is in production, it definitely needs to maintain and or improve performance...
We have idea about how to reproduce but using deterministic training mode is very slow as it loses some optimisations.