All results should be published, some should be celebrated.
All results should be published, some should be celebrated.
Some people publish fantastic papers without data nor code. Sometimes annoying, other times a complete waste of everyone's time.
In a world where they are decoupled, you'd only end up reading it because somebody you trust endorsed it (no reason for that to coincide with the publisher) or because you went searching for it despite its lack of endorsement--in which case you probably know what you're doing.
I was specifically talking about peer-reviewed papers and how bad some of them are. I'm working in the space of parallelization and optimization and a lot of papers out there just implement any given algorithm on any hardware of their choice. They describe vaguely what they have done, claim to be faster than any other papers and call it a day.
That's worthless in my opinion. For all we know they could have a bug in their code and that's why it's fast. Some may even lie, who knows without code or data.
But oh well, I guess it's a step towards graduation for them. So at least they got something for their time (and presumably they have that cool, fast implementation they need for a project and some useful experience, but the paper: worthless).
While the resemblance of my A chart with their B chart might be a bit subjective, I have a reproducible build of my chart and its labeling that I can trace all the way back to the source data (via nix). It might not be perfect, but at least there's an artifact available for scrutiny... at all. The published version though, it's just a PDF. I can't see their code so that I can point them at the bug that I suspect exists in it. Their data is available on NCBI, but not their code.
So yeah, I hear you re: being a bit disappointed with the reproducibility of the computational aspects of these papers. It's somewhat understandable for replication to be difficult in places like psychology or chemistry... but if the work is being done in silico, it should be a matter of two or three commands to re-do it in silico. And if that's not true, we need to fix the tooling until it is.
I'd prefer to have the raw data, intermediate artifacts, and their computational relationships, all made available as soon as the researcher has an inkling that they might be worth presenting to someone. Then, when it's time to celebrate those results, they can just reference things that have been publicly searchable and verifiable for a while now. That way computational verification can happen before the paper is even written, and we don't even have to bother writing prose around what turned out to be bogus results.