Comment by DavidSJ | Hacker News Reader

Comment by DavidSJ | Hacker News Reader

Parent Full thread

To reduce the occurrence of future similar programming errors, the Johns Hopkins Biostatistics Center has instituted a new standard operating procedure for checking randomization assignment to be followed in all trial analyses. To ensure that the group assignment used in any of the trial analyses is correct, a verification process will be included at the beginning and end of each analysis program. This process is intended to confirm that the group assignment separately provided by the trial team matches the group assignment used in the analysis program. The matching confirmation is reviewed by a second biostatistician/analyst before its use in the results.

I don't know what software quality control is already in place at this organization, but this corrective measure seems on its face wholly inadequate to me: they're just preventing a recurrence of the same exact problem, rather than the much broader class of problems due to programming errors. Do they have a code review process in place?

This speaks to a larger issue: if you write software for manipulating data as part of the production of a scientific paper, then the source code should be available for review as an attachment to that paper, and review of said code should be part of the peer review process in any reputable journal. Professional software engineers write bugs all the time that invalidate the correctness of their programs, never mind individuals whose primary job is research, not software.

>This speaks to a larger issue: if you write software for manipulating data as part of the production of a scientific paper, then the source code should be available for review as an attachment to that paper, and review of said code should be part of the peer review process in any reputable journal. Professional software engineers write bugs all the time that invalidate the correctness of their programs, never mind individuals whose primary job is research, not software.

Completely agreed, the source should be open (ideally FOSS) - but also, the software development should be conducted properly too. On a practical level, using version control and a code review mechanism (e.g. GitHub PRs) within the research group equivalent in rigorousness to what you'd see at a good practice software development shop in industry.

Clinical decisions are made off the back evidence published in peer-reviewed, respected journals. It would seem to me that serious software errors in this domain have the capability to contribute to grave patient consequences. Much more serious consequences than if I introduce a bug into a client project.

> On a practical level, using version control and a code review mechanism (e.g. GitHub PRs) within the research group equivalent in rigorousness to what you'd see at a good practice software development shop in industry.

Let us (professional s/w engineers) not pat ourselves in the back by confusing standard industry practices with 'rigorousness'. Rigorousness would be formal verification and proofs. How many s/w engineers can do that? How much will it slow down the development speed?

There’s a tradeoff between rigor and output. Except in toy or very niche problems, I’m not aware of formal verification striking a good balance between the two, although perhaps I’m just ignorant. But it seems to me that code review and some unit and functional tests are usually a clear win.

Right, that's what I meant by code review in the first paragraph: internal review within the research group.

Agreed also that they should use version control and other standard practices, etc.

The way many clinical research groups are structured (unfortunately) precludes code review, and to some extent, version control. Often for projects like the one referenced in the OP you'll have just one statistical programmer working with a PhD-level biostatistician in addition to the MD investigators. The biostatistician guides the programmer through the statistical methods to use and steers the overall study design, but otherwise they never see the underlying code. There are some exceptions, but in many cases, the programmer ends up the only person seeing the code. A lot of this has to do with how funding is structured—grants are written assuming one FTE programmer, and something like 0.05 FTE for the biostatistician. It's hard to convince funding agencies that you need more than one FTE programmer in many cases; on top of that, farming out the biostatistican's time in small chunks like that splits their attention between dozens of projects, which precludes them engaging with any one of those projects in depth.

So, there's literally nobody else on these projects who could conduct a code review. This also sort of provides a disincentive towards using VCS even though it's so obviously a good idea if you're the only person contributing code—I've talked with programmers about this before and the response is "why bother?" unfortunately.

> Right, that's what I meant by code review in the first paragraph: internal review within the research group.

Ah, I took it as code review during the peer review process (i.e. from reviewers). Unsure how realistic this would be.

> Code review during the peer review process

Unfortunately, this would make the already laborious process of peer review even longer and require more work. Given the reality of academia today (publish vs perish), not to mention the added work of preparing even well structured, version controlled code (which is not, in my experience common) for publishing, most researchers would not opt-in (or support) something which would make publishing more difficult.

Maybe this sounds harsh, but:

Who cares? If researchers are producing crap to get published, why should we mind if they stop producing that crap when journals raise the standards of publication?

That’s what I was referring to in my second paragraph. :)

Reply on news.ycombinator.com