Hacker News Reader

Comment by OJFord | Hacker News Reader

> It’s a weird thing about academics — they like to write papers and “share ideas”, but it’s very hard to get source code from them.

I suppose at least part of the reason, especially for things like crypto and statistics, is that they don't want to get bogged down in plausibly-correct claims of fatal flaws, and be perhaps repeatedly (over years hence) put in a defensive position of having to prove the criticism incorrect or have everyone assume the work was wrong?

I can sort of understand that. The same's true for the papers themselves of course, but I can see it being more annoying for code. ('But it's never going to actually be in that state', etc.)

View on HN

You're probably right. And assuming you are, then I think maybe the core problem may be that academics even tolerate the petty undercutting of reputation based not on the merit of the big ideas, but rather minor flaws in the proofs of concept.

There's also a problem in allowing papers to get away with making arbitrary claims that are by definition irreproducible. Every paper always includes some section about how many gates a design took up and how fast it ran -- but there aren't entirely consistent standards for reporting these things; it's often too easy to hide significant setup costs when running benchmarks.

This leads to an incentive to overstate results, because without any threat of checks it would be disadvantageous to not cherry-pick the absolute best numbers you could get through a peer review committee.

I get the desire to escape petty criticism: but it seems like a weak argument against the myriad of reproducibility and integrity hazards such a lack of transparency begets, especially for a line of work that ostensibly distinguishes itself on high standards for merit and integrity.

It also would have been nice to just receive a polite response of the form of "sorry, I don't support this anymore, I've moved on..." from anyone I had reached out to. Before starting on this project, I emailed four or five groups that had all published various papers on Curve25519 accelerators hoping to get even a broken non-compiling repo that could have helped me sanity-check my understanding of the thornier concepts presented in the papers.

Not even a single response...hence the motivation to write up notes and share them. Seems a waste of funding to have so many implementations, but no way to actually use any of them...

Actually thinking about it, another aspect is almost certainly 'commercial sensitivity' for want of a better phrase? The university will own the copyright; they might have a policy against just handing it out when they might want to be able to later decide to commercialise it (because it's some whizz bang faster than everyone else's 25519 accelerator)?

I spent my first internship at Arm benchmarking various libraries' performance on Cortex-M. Arm subsequently acquired PolarSSL, now MbedTLS (I don't recall my results - no claims here, about PolarSSL nor the influence of my lowly undergrad intern work!) for I assume some non-trivial amount. You could imagine a similar company acquiring WhizzBang25519AccelCorp from the university but perhaps less likely/for less if there's a convenient WhizzBunnie25519AccelOSS re-implementation?

You might be right about that concern, however, a lot of this work is done through public funding. Seems a little bit like double-dipping to get the public to pay for you to make IP that you then later sell to the highest bidder, with no obligation to share to the public.

Of course, if the professor was say, the NXP chair for crypto core excellence and they endowed her department with the understanding that all IP would belong exclusively to NXP, then sure, that would make more sense. I did not see anything like that, though, in the research groups' bio.

One reason you sometimes don't get responses on this sort of thing is that the claims in the paper are sometimes just fake.

Not totally fake-- e.g. they'll do part of the work so that they feel like they can extrapolate to the rest. Like they might implement a field multiply and benchmark it, then just count how many of those they need for the whole circuit.

Then no one wants to waste the column inches explaining what they actually did or dealing with reviewers that want to nitpick it ... at least not if they can help it. I know I've personally regretted including experimental details for that reason. ("oh great, a reviewer wants me to re-run 20 CPU years of number crunching because they've taken issue with some irrelevant detail that I could have just left out. Fine.")

The "more approximate than they let on" doesn't just come from sketchy and obscure research groups but this kind of work happens under well known names too. If you read a lot of the lit you can get an impression on which groups/people are actually competent implementers because they publish them or at least also solve sub-problems that you'd only encounter when creating a full implementation.

There is a lot of pressure to go this route because of time constraints and because the engineering to actually implement this stuff is often too far outside the authors expertise-- the people who really know about implementations can get highly paid in industry and many do.

For anyone who isn't faking their research this should be all the more incentive to publish your code! But so far it seems the incentives haven't lined up that way yet.

Donald Knuth once wrote a paper with some source code.

He warned his readers about just copying the code, perhaps half jokingly.

He said, approximately, "this code has been proven correct, but not tested."

Are you saying this based on experience? My assumption was that it was a number of factors.

1. Researchers are often weak coders, and they may be self conscious about their code.

2. My own experience is that researcher code is totally undocument, or broken. I've repeatedly found that the code linked to from a paper has no straightforward way to be built, or appears to have compiler errors etc.

3. It's more work, and researchers have the incentive to put out a compelling paper, not to put out code (because we don't hold them to that standard).

My own experiences have led to me to mostly dismiss papers where I can't find the code or can't get it to build.

Given all that, it's understandable that it's not released and completely reasonable - a support structure is needed if the code should be released.

I'm setting the bar super low. A `./configure` and a readme with basic info about the project and how to build/run it would be quite exceptional. I don't need them to be triaging github issues and merging in PRs.

> Are you saying this based on experience?

No, just observation & supposition.

> My assumption was that it was a number of factors.

I'm sure! I did say I suppose part of the reason.

Didn't mean to imply any experience/authority, OP just made me think about it. Sorry if it sounded like that.

Oh, my question was genuine, I wasn't trying to be like "you DONT have experience". I was just curious.

Rereading my post it sounds like I'm trying to disagree, I'm no expert either.

In a field like crypto you also have the concern that someone might copy your implementation and actually use it - this is fine if your specialty is crypto implementation (resistant to side-channel attacks and so on), but if your specialty is instead coming up with faster algorithm, your implementation is going to be much more proof-of-concept. In the theme of “don’t write your own crypto” is “don’t release your own crypto if you look authoritative (Eg a professor) and are not absolutely sure of it “

In the biological sciences there is a huge push for the code to be released alongside the paper. Many publications (but not enough yet) include a complete pipeline in a workflow language that downloads, processes the data and generates the figures used in the paper.

What’s the language de jour typically leveraged to declare these workflows?

Since GP mentions biology, I would guess R is the first choice.

Nextflow and Snakemake are quite common. I prefer the later as I have a lot of Python experience.

Reply on news.ycombinator.com