For those here that work on ML or data science teams in industry, do you use something analogous to preregistration?
I think it's relatively common for practitioners to have an idea for some change to an existing model or system, be that a structural change, new feature, some preprocessing step, etc, develop it by iteratively making changes and running some pipeline, and stop when it looks like a measurable improvement has been achieved. Cut a PR, put in some graphs of the change in metrics of interest, merge, and start the process again. Any sequence of negative results encountered on the way are quietly presumed to be because something in the code or the setup wasn't right yet -- and often there were clear problems which get fixed along the way. But as a result, those negative outcomes often aren't kept or reexamined.