It is a real issue. The programming languages and software engineering communities have been working on evaluating software artifacts for a few years now:
This doesn't address open access, but it does make sure the software is usable by a non-author to reproduce the results of the paper.