Software updates: the “unknown unknown” of the replication crisis

The problem is you can't document everything while also trying to automate everything. It's always a balance of figuring out which information is most important, and if 'oneself' were to attempt to construct a model that represents everything, 'oneself' would simply be stuck in a catatonic state of analysis paralysis, in perpetuity and invariably, until the heat death of the universe.

Yes, the onus is on the user, but who is the 'user' in the industry of development? How do you maintain structure in records for information systems that are fundamentally, at core, completely opened%? I believe this problem maps back to the science of collecting and verifying the veracity of information, up to isomorphism (quite literally, this is the definition of the graph isomorphism problem).

% The intrinsic nature of science and discovery is 'we discover new things because we assume there is more to know (~ we don't know everything)'. Therefore, all components of such a system are subject to change. There is nothing that can be said to be permanent in knowledge discovery systems where the premise of discovery rests on 'assertions' that can not be provably demonstrated to be factual AND absolute.

Recording version, etc. is fine, but what use is it if it's not actually possible to install that software to reproduce whatever results were obtained? (This also extends to hardware, btw., but VMs can probably attain enough fidelity for that to not be an issue as long as the physical media are still readable.)

Example: Reasearch done on Windows XP at some point in time past SP3. Unless you're prepared well ahead of EoL on Windows XP there's really no way to reproduce that environment.

(I'll definitely grant that it's probably unlikely that the obscure details of the OS have that much impact on whatever results were obtained by programs running atop it, but it's at least moderately plausible.)

It's like when fixing a bug in library code breaks application code. Usually it's because there was some undefined behavior in the library - which wasn't part of the contract - which the application (knowingly or unknowingly) relied upon, and then the updated version produces a different undefined behavior.

It's a fair question.

Here's a possible recipe to get such variations in estimates: (1) an estimator that does not really match the distribution of the dependent variable; (2) small sample sizes with insufficiently well-handled influential observations; (3) (robust) standard error corrections leading to disproportionate confidence intervals; and (4) limited work on diagnostics, which is another way to make all of the previous points.

The points above can be used to 'take down' many papers published in journals like the one in which the incident happened, but you can also take a more charitable view and rescue most of those papers by claiming, for instance, that statistical significance should not govern (and even less govern alone) over the identification of the data generation process.

My conclusion is therefore: yes, algorithmic variation makes those results suspect, but on a single dimension that should probably not stand as the most important one in assessing those results in the first place.

Edited: syntax.

You'd think a simple percentage grade would have meaning, but it doesn't. Often times federal and state rules alter how data is to be "perceived" and therefore, analayzed and reported on.

There are a few major players in the data collection and analysis game for public school system, and they all not only have their own way of determining skill level (eg, reading skill level) with their own testing systems, they also analyze their custom data with their own algorithms. Thus leading to quite a variation in their ultimate recommendations/conclusions between vendors.

Then, take into account some schools have many students with English as a second language, and based on how the state views that school, it may be analyzed as "all equal" or take into account the language barriers.

Then mix in that you have to analyze students over multiple years, with students entering and leaving districts, sometimes with data vendors changing, etc... Then there is state and federal laws mixing with multiple vendors data collection _and_ analysis.

So the next time you hear a particular school district is under/over performing based on these vendor's analysis (especially related to budgets), keep in mind how variant the actual reality is to what is being reported.

Maybe every complex system has the same core problems?

This is what developers deal with all the time, so this is obvious. Code should work this way, but it doesn't always. Yes, we can all make heroes of ourselves by thinking of ourselves as saviors for those edge cases of catastrophic failure, but we don't, because we are aware, that bugs happen - they are a mathematical certainty assuming one doesn't have access to all information, including every single use case of a program in the future.