"if you roll your own crypto, and it breaks, you will look incompetent." - tptacek to cperciva, 313 days ago.
http://news.ycombinator.com/item?id=1183757
(not that I know anything about crypto)
http://news.ycombinator.com/item?id=1183757
(not that I know anything about crypto)
But yeah, I hope it's obvious that I see this as a very strong vindication for my argument that generalist devs shouldn't build crypto. At all, ever. Use TLS for data in motion; use PGP for data at rest. Systems much bigger and heavier than yours have gotten away with this.
Generalist devs shouldn't build crypto. Expert devs shouldn't build crypto without review.
As for academics... a lot of us aren't as hopeless as some of the trash that appears in conferences and journals might appear. :)
I was surprised that Colin's solution is to personally re-review his code. Good writers know--don't rely on yourself for proofreading. Usually the mental lapse that caused the problem will manifest itself during your review as well.
Disclaimer: I am not a tarsnap user.
But please, go ahead and give the code another read. :-)
Edit: Totally willing to continue burning karma on this comment if the HN community continues to decide vote it down. I've tried reviewing my own code in a different state of intoxication than when I wrote it and I'm not joking that it can help. I'm still trying to pull resources together for a study on the benefit of different mindframes for peer review. We haven't tried alcohol yet, but frankly it wouldn't be a half bad idea if we could get anyone not to laugh too loudly at the proposal.
"they are wont to deliberate when drinking hard about the most important of their affairs, and whatsoever conclusion has pleased them in their deliberation, this on the next day, when they are sober, the master of the house in which they happen to be when they deliberate lays before them for discussion: and if it pleases them when they are sober also, they adopt it, but if it does not please them, they let it go: and that on which they have had the first deliberation when they are sober, they consider again when they are drinking."
http://www.gutenberg.org/cache/epub/2707/pg2707.txt
I also agree with you in general, that checking things in different mental states is a good practice. With alcohol, I suspect the benefit is outweighed by the difficulty of spotting bugs when drunk -- but who knows?
I suppose I could try reviewing code in both caffeinated and decaffeinated states, but being decaffeinated gives me enough of a headache that I don't think I'd be much use that way.
I know that I find more mistakes in my code when time reveals the code as it is rather than as it was intended. But I can't say this makes me good enough at proofreading myself. What about the code I've conceived and written in ignorance?
You count how many bugs you find, then you count how many bugs other reviewers find.
That's useful advice, if you need and _want_ the guarantees given by TLS or PGP. If you have other needs then a look at, say, off-the-record messaging may be useful.
OTR is just the first example I could think of, that gives different guarantees than most normal cryptosystems. I don't particularly recommend it for anything apart from instant messaging. And I wouldn't recommend implementing your own.
If I speak to you in private (and we know each other), you can be sure you are speaking to me, but you won't be able to proof to any third party anything I said. OTR can give you something like that. PGP can't.
For most application you will be well served with PGP or TLS. But be aware of what baggage they bring. For some areas losing deniability via PGP can be worse than plain text.
This is a moot point, because most systems would never care enough to intricately position all their features just-so to compose OTR-like features out of PGP primitives. What they need is to be able to encrypt anything without implementing trivially exploitable crypto vulnerabilities that were discovered and solved decades ago.
This is a textbook case of everyone's good being strangled by someone's opinion of the perfect.
Me either, and while I chuckled, I think cperciva is one of the better qualified people to be implementing crypto.
cperciva is extremely qualified to implement crypto, but not without review. I think it is wise that he has implemented a bug bounty procedure. He should make sure it applies to unreleased versions too so maybe someone will put an RSS feed of his SCM checkins into their RSS reader and try and catch bugs as he's making them. :)
It would be nice if he had the money to spring for paying someone else to look at all his changes, but alas... that stuff is expensive!
No professional is going to undertake a review on spec. The demand for software security is too high; most of us have our pick of interesting projects that will pay whether we find something or not. We're not unique in that respect; top iPhone developers won't work for you on spec either, not because spec is evil, but because the economics don't work.
Furthermore, you can pay $1000 for XSS bugs and random memory corruption flaws in browsers because fuzzers can find them, because they're luck-of-the-draw findings, and because people are hammering those things whether you pay them or not. But $1000 doesn't pay for a day of qualified review, and no qualified reviewer would suggest less than two weeks for something like Tarsnap.
However, since Colin presumably doesn't want to raise his prices to pay for actual review, it is encouraging that he is at least going with bug bounties. These, at the very least, gives us a good excuse to assign them as fun things to do for graduate students with some hope that one will want to procrastinate so hard that they will actually look at the code.
Also I think any reviewer who wanted to get paid would not start with Colin's code as an easy place to find bugs.
It will be interesting to see how close you can manage to get something resembling good review on a budget. Hopefully other people who are in similar low margin code businesses will keep an eye on your experiment to see how it works out.
Thanks for being so open about how you're trying to make things work. I hope you'll be publishing all the awarded bounties? (I suppose I should just wait for your follow-up entry.)
Speaking as a Tarsnap user, he ought to. The service is seriously underpriced right now.
In the meantime, the bug bounty + very qualified developer strategy seems like a reasonably sensible option while the service is presumably, still in its growth phase. I guess we'll find out.