"no one should ever rely on a black box in this field"
This is deeply hypocritical. When someone publishes results generated by code, but don't publish the code, they are asking all readers to rely on the output of a black box.
This is deeply hypocritical. When someone publishes results generated by code, but don't publish the code, they are asking all readers to rely on the output of a black box.
But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer, if you disagree, you implement a different methodology, which necessitates new code anyways, and publish your contra-finding.
We usually converge on answers over years of varying different methods and attempts. No one in their right mind reads a paper and says "well, they found out the sun is actually in M31, I believe that now" They look at the culmination of the literature (fun fact, we still aren't positive exactly where the sun is, or how fast it's moving) and I suppose that is the black box.
What I was really referring to though was students who walk up to me and say they gaussian smoothed a sample, and have no idea what I mean when I ask if 3 sigma outliers were used in the fit or trashed. They just used some gaussSmooth algorithm and may or may not know what a gaussian even is.
But to address the main point - there are two separate things to consider when assessing a paper's methods. First is the methodology, which as you say is always included in the paper. But secondly, and equally importantly, is the implementation.
Agreeing with the methodology does not make me confident in the results. Someone wrote (probably) a lot of code to generate the analysis, and the likelihood that it contains bugs is high. They may or may not affect the outcome. Without seeing the code, I'm not going to trust the results.
Of course, I don't expect to read the source code of every analysis, but if it's open to scrutiny by the community, and the results are of any importance, it will be validated.
The problem is actually worse in many experimental methods, where the results rely completely on the practitioner having done exactly what they say they did and done it correctly. No source code to publish there, but that doesn't excuse not publishing analytical code when it is available.
> But they are presenting the methodology they claim went into the code and allowing you to agree or disagree with the methodology. If you agree, you take their answer--
No. And this is the whole point. You can agree with the methodology, but as long as you haven't inspected the code, you cannot just accept they actually implemented it right!
In fact, chances are, they probably didn't. The best programmers white buggy and incorrect code. And from what I've seen in the field of physics, scientific code is everything but an exception to that. I'd be surprised if it was very different for astrophysics.
So by publishing the methodology and the results, but not the code, they put up a nice show. But that's it, not reproducible. So all that's really brought to the table is the methodology and some "say-so" results that nobody can check are accurate.
But it is quite common for someone to take the same problem, collect their own data and run their own analysis (both steps can be completely different from the original statement-- collecting data of a different type, from a different instrument, etc. Running an analysis of a different paradigm, running the same analysis to a different precision, running the same analysis via a different algorithm, etc.).
If you read the arxiv on a daily basis you can see huge academic arguments unfolding over the course of months and years.
There seems to be this idea that the conversation goes: "Yo I found this hypervelocity star" "Dope, let's move on"
It's actually more like: "Yo I found this hypervelocity star" "Nope, I got spectra and you're wrong" "Well I got ultra high-res spectra and I think he's right" "Actually all of you are forgetting asymmetric drift, this is just a geometry problem, l2angles" "Hey, I sit in my basement and play with MOND, it might help" "My simulations show something completely different"
Authors are called out and proven right or wrong on a daily basis, even if we can't watch them code over the shoulder. I actually think that's one of the beauties of it-- most of our methods are invented by trying to prove or disprove something in a new way.
The monster codes like GADGET (which ran the millenium sim everyone's seen) are usually made public after ~5 years of proprietariness.
Haha, that actually sounds almost exactly like the Stone Soup parable :)