I also think code used to produce results should be published as well.
Honest to god and on the grave of my mother this happened to a friend. He was going over some old Fortran 88 code, filled with GOTO statements. The code, at best, was a rat's nest. Only a deep and long fight could get it into your brain. At about 3 am after a long day in the lab, she finally gets to a line in the code that says GOTO LINE 12345 with the comment 'HAHA MADE YOU LOOK' . Her boss bought her a new computer after she threw that one off the roof.
That experience is universal with Biologist code.
That's not actually the point. The point is reproducibility. If you write your own code and get a different result, why? If the reason is the code then having the other code, no matter how terrible it is, allows you to figure out what the difference is. And then somebody is wrong and you know why and can publish that result.
The end result is that empires are built, reproducibility is (paradoxically) harmed, and we're someday going to end up finding out that some big, high-profile projects were built on pillars of sand.
From what I've seen when supporting EEs? People who are that much smarter than I am don't have this limitation. They can write thousands of lines of spaghetti perl and bash and as long as nobody touches the damn thing, it works fine. God help you, though, if you make a change.
But... there is an established job role for this; I mean, you don't want to make your EEs sysadmin their own LSF cluster, either.
Or need to add functionality. Or, just wait a year.
I think sometimes you just get so caught up in your field of expertise that you miss some of the tools that could drastically help you entirely. You have to wonder how much of an efficiency drain that adds up to over the course of even just individual research projects. And that's before you get to bad code.
I got access to the source code, and the super complicated algorithm added almost nothing to the results, the glossed over / hand waved past data normalization worked so well that there wasn't a need for any further classification. This paper was pretty well received and cited, despite it basically not work.
I think a central hub for public comments, with source, and available PDF would be incredibly useful for CS as a field. For most papers I had access to, to get the source I'd have to go through at least one person. So I tried for quite a while to replicate the result before even trying to get access. If it was available, on say github, I'd have grabbed the source code as a resource to understand the paper, and with something like publicly available commenting, I think it'd have been a non-issue.
The problem is, I can see how such a system would be incredibly constructive to the field and the community, but it could be a liability to the authors, and would make publishers meaningless, so I doubt it happens anytime soon.
Especially because if you are forced to publish your code, it will be better.
Academic code sometimes only has been made to work on one machine (the author's). Someone reproducing the results should still have access to the source, even if they have to hack it to work on their system.
;)
But seriously.
The CRAPL: An academic-strength open source license http://matt.might.net/articles/crapl/
To be fair, a lot of people do publish their working code with the paper. It helps people like me understand how to do the work.
Bad code is fine, good code is better, great code is great. Stick to fine and I can do my job and you can write papers knowing that your work will be used by many.
A much bigger problem is that grantsmanship strongly incentivizes against verifying your methodology and being diligent in your construction of null hypotheses.
Well, if you're embarrassed by your terrible handwriting, maybe it's a time to put some effort to improve it.
Really, a lot of the comments to this and the downvotes seem to betray a lack of understanding of what cs research is like, which is understandable I guess. CS research is not like creating physics experiments, there are methods, techniques, and algorithms. When there is an experimental result, CS researchers do not treat those as sacrosanct and do have an informed skepticism but we are generally able to determine and discern the bulk is what is important from the content of the paper itself, in most cases.
Yeah, if you look at the experimental results section from papers in SOSP, you could come up with all kinds of complaints probably all day long but there is more to it and the community than is being taken for granted here.