Link to relevant announcement email:
https://lists.zx2c4.com/pipermail/wireguard/2021-March/00649...
There's also Jason's reply to apparently not-nice feedback of someone from NetGate: https://lists.zx2c4.com/pipermail/wireguard/2021-March/00649...
There's also Jason's reply to apparently not-nice feedback of someone from NetGate: https://lists.zx2c4.com/pipermail/wireguard/2021-March/00649...
According to the article linked and the info here in that email you linked this is my conclusion:
* Netgate tried to ship flawed code that has multiple security issues.
* Jason Donenfeld, one of the lead Wireguard developers, went out of his way to work on rewriting it to be better in time for the 13.0 release of FreeBSD
* This Netgate employee is angry that they weren't able to ship their bad code and starts throwing accusations of a smear campaign.
Am I understanding what happened correctly? Because it really makes this Firewall/Router look really bad.
The first step was assessing the current state of the code the previous developer had dumped into the tree. It was not pretty. I imagined strange Internet voices jeering, “this is what gives C a bad name!” There were random sleeps added to “fix” race conditions, validation functions that just returned true, catastrophic cryptographic vulnerabilities, whole parts of the protocol unimplemented, kernel panics, security bypasses, overflows, random printf statements deep in crypto code, the most spectacular buffer overflows, and the whole litany of awful things that go wrong when people aren’t careful when they write C. Or, more simply, it seems typical of what happens when code ships that wasn’t meant to. It was essentially an incomplete half-baked implementation – nothing close to something anybody would want on a production machine. Matt had to talk me out of just insisting they pull the code entirely, and rework it more slowly and carefully for the next release cycle.
I can understand being upset if that's how you're portrayed publicly.
They're not some random person but are representing their company with their code.
If there was a security exploit with their Wireguard implementation, would Netgate get blamed or Wireguard?
Jason should have informed Netgate the quality of the code is shit in private and FreeBSD dev should have told Netgate will not be shipping any of it in Rel 13.
It is then up to Netgate to decide What to do with their Rel 2.5
To play devil's advocate: Netgate isn't paying Jason, and they're taking his open source code to create a proprietary commercial project. I'd say Jason owes them exactly nothing in the way of courtesy or consideration. Could he have been more polite for the sake of being polite and community goodwill? Probably.
I am not sure if that is the case. Netgate seems to have used their old crappy sponsored work for their Pfsense.
That is judging from the two pieces of information here. Jason doesn't need to be of consideration for Netgate. There could be other communication we dont know about. I can certainly understand why Scott is frustrated.
Their sponsored work was based off of the Linux and OpenBSD code that Jason and others wrote. And even if it didn't utilize that code, you literally can't write a wireguard client without building on Jason's work.
(And there is, of course, the question of whether Netgate is in any way "professional" by building on top of an open project and not following its norms.)
From what has been said it's not like they found and fixed a subtle and cryptic vulnerability in an otherwise reasonable implementation and then failed to disclose it properly. It's more like they turned over a rock and found a murder victim. The guy from Netgate is also coming across as very inward looking and seems to assume everyone else's motivations are also purely selfish (referring to his comment implying a "shower of contracts" they might receive for the publicity). His focus should be on how to prevent this mistake from happening in future.
If I'm Jason and I offer my help (for free), they don't take me up on my offer, then try to release code that would make my baby look quite ugly, I would probably also have a pretty severe reaction.
Could Jason have been slightly more professional? Absolutely. But we're all human and I can't entirely blame him, I'm sure he was frustrated that he offered to help multiple times and they both didn't take him up on the offer, and tried to release a hatchet job with his name (indirectly) attached to it.
So it is first and foremost on Netgate, if Jason is right but right after that it is on the other responsible FreeBSD co-developers. I mean having so obviously bad code in any kernel of a modern operating system release would be really, really bad. There are many people and companies dependent on it that cannot really influence anything but have to endure the consequences either way. I mean, if you buy a storage appliance, a router or a firewall you trust the quality of the product to a degree and cannot really audit much even if you had the skill. You have to take the word for it and make some reasonable accomodations. No insurance is going the replace the full damage due to lost data to an attacker or a bug. Peoples lives sometimes indirectly depend on the full chain of competence and no insurance can resurrect the dead or right the good name of anybody. Remember, most of the time when you have to update anything for security reasons, somebody didn't understand the system fully or just plain messed up. The only exception is when the problem or times / requirements have changed (e.g. the computers got so fast, we have to transition to longer keys/ passwords whatever).
So yeah, if Jason's account is accurate it is bad the code landed in the codebase at all and raises questions about the quality and security of FreeBSD. I mean, it is code directly meant for a secure-as-possible VPN and something that often directly interacts with the open internet. Surely such code should experience extra scrutiny.
From the short personal interaction with Jason he came across as quite thoughtful and knowledgeable. Over the years, he and his supporters were able to convince many not so easy to convince people about the quality of Wireguard and some of its implementations. He and the supporters have shown a long term commitment and I am for these reasons inclined to trust Jason's judgement as well.
Until he issues a public apology for his actions, I'll refer to him as Kip. Changing your name to run from the google searches is completely understandable, and I support second chances, but you need to show a bit of remorse IMO.
https://abcnews.go.com/US/exclusive-landlord-hell-defends-te...
... no, no, there's no reason to hold actions like that over someone's head. It's entirely praiseworthy and I'm sure it's really easy to cooperate with such an upstanding character.
Oh, I don't entirely blame him. I just partially blame him for not seeing the obvious way this could devolve into a problem, even if it would (justifiably) seem unlikely to go to this level so fast. That is, he shouldn't be surprised there was a problem with what he said, although the scope of the problem is a bit more than I think most would expect.
Professionalism isn't just about making others feel good, it's about optimizing for useful outcomes, which includes covering yourself. Not taking care with your words is just like not taking care with your code. Sometimes there's a weird interaction and things go boom.
Even if all of the above is true, it reads like an elaborate insult. And that's fine if that what the author set out to do for some reason. Pretending it wasn't after the fact isn't being honest, in my opinion.
A more professional and neutral announcement could just talk about code that needs to be refactored due to some incompleteness and vulnerabilities.
To a much greater extent than in other security protocols, implementation security is a goal of WireGuard. The protocol itself was designed to support secure kernel implementations; for instance, it's designed in such a way as to not require on-demand dynamic memory allocation.
It's part of the premise of the security model of WireGuard that it has secure kernel implementations. If you're building a kernel WireGuard implementation for a major open source OS without taking advantage of the WireGuard implementation design concepts, you're not really building WireGuard; you're building a compatible fork and calling it "WireGuard".
The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Instead, the WireGuard project seems to have been put into a position where they had to scramble to fix up an implementation that was being pushed into FreeBSD, as WireGuard qua WireGuard. I can imagine that being a frustrating experience. It certainly didn't generate the most political response ever, but I think you'd be reaching to call it a deliberate insult.
My read on it wasn't that it was an elaborate insult, but more that it was far more denigrating than it needed to be, if he was trying to be professional. That doesn't mean it was purposeful, sometimes people just don't really associate the statements they make with how it may be perceived.
I think it could have been communicated clearly and succinctly with something along the lines of: "The first step was assessing the current state of the code the previous developer had dumped into the tree. We noticed some quality problems, some unimplemented protocol sections and more concerning, security issues with the code. Given these issues, we considered asking they remove the code, but instead Matt convinced me that we should rework it slowly and carefully for the next release cycle."
Notably, I think omission of the following inflammatory statements would have prevented a lot of problems:
- "It was not pretty."
- "I imagined strange Internet voices jeering, “this is what gives C a bad name!”"
- "the most spectacular buffer overflows"
- "the whole litany of awful things that go wrong when people aren’t careful when they write C."
Whether those entirely subjective statements are accurate, they are not the things you say about someone else's work output when you expect a useful dialogue with them, which is exactly why they are considered unprofessional.
I'm not defending Netgate's code here, or even the vehemence of their reaction and how they went about it, but merely noting that not only can I see how it devolved into this, I would go so far as to say it's obvious that this is why that type of language is avoided by most people trying to work professionally. Jason wrote some very unkind things, and Netgate blew up about it. There's enough blame here that they can both share some.
> The "ask" here from Jason was for everyone to slow their roll, take the flawed WireGuard implementation out of the tree, and give everyone a chance to make it more resilient. Considering the amount of work Jason had to go through to get WireGuard into the Linux tree, that seems like a very reasonable request.
Err, wasn't that actually not the ask, because he thought they wouldn't do so, so instead they worked it over in a short time-frame, only for it then to be removed when this argument broke out and it came to light?
Hopefully I accurately expressed that as what I was trying to convey. I don't think Jason is close to even half the problem in this case, just the small spark that allowed it to continue and explode (continue because is started with a substandard implementation to begin with). At the same time, he's also the one easier to critique constructively because the other party is hard to relate to (I'm not one to jump to conspiracy theories about implicit efforts to defame).
When you're talking about replacing and rewriting the implementation on the eve of release, you better have a good reason for doing so. Stuffing a rewrite of security critical code into the kernel at the last minute is a big red flag. The main question that immediately comes up in that context is, "how is it possible that having a last minute rewrite would be better than the code that was there before? You've only looked at this for a week." And that's a really good and important question.
That much code churn is not something I wanted when I set out to get started with this, but it's ultimately where things wound up. Why? For exactly the reasons I described in my email. The idea wasn't to be _insulting_, but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite. I see how perceptions could view that instead as denigrating, but that wasn't really the motivation. And it's not as though anybody really is rushing to defend that code either; it doesn't take a lot to look at that and make up your mind that it was probably unfinished stuff, not coded with much love, that was committed prematurely.
It also had the, I think, positive effect of leading to more scrutiny of the review process. A few people have piped up and mentioned to me that their concerns during that review weren't addressed. And as a consequence of everything, all of the code, including the rewrite, is being removed from FreeBSD until it can be carefully examined and completed, which is really the best of conclusions.
Sure, I didn't really interpret it as you attempting to be insulting, more that you were accidentally insulting through your explanation of what you found.
> but rather to accurately and vividly describe the state of the code, as a motivating factor for the rewrite
Sure, but is any of that really needed beyond "there were numerous security problems we had to address"? When talking about shipping crypto, I think most involved would agree not shipping it is better than shipping something possibly exploitable.
I think the core of what I was trying to express is that words should be crafted with care when expected to be read in a public forum like this, just like any code expected to be used by many should be crafted with care. For the same reason it's useful to remove quadratic algorithms from places where the input is somewhat not entirely vetted, it's useful to take care with words to reduce the chance of misinterpretation.
That doesn't mean scour your statements for the smallest possible misinterpretation, but there's a lot of room to improve things like "I imagined strange Internet voices jeering, “this is what gives C a bad name!”" while still expressing your point constructively. The low hanging fruit is easy to pick, so you might as well pick it.
To be clear, I feel for you with regards to this situation. Nobody really expects weird accusations like you got from simple emails, and that's on Netgate, but a less extreme response that also publicly notes the soured relationship would also be a negative outcome from this in my opinion, if one of lower magnitude.
Unfortunately this type of behaviour is par for the course for Netgate/pfSense.
Btw. thank you (and all the people that probably help you here and there) for Wireguard. I put a lot of trust into You/ Wireguard and I am hardly the only one. So I am all for quality and security even if it takes as long as it takes. It is really a huge progress we have a useable and _simple_ VPN protocol and widespread implementations of it.
However, on a professional level, when a core maintainer of a protocol tells you your code is bad, it's time to sit down, eat some humble pie, and start taking notes. I think that's especially true if you're offering help or are willing to address online communities with positive messaging to help them save face.
I think you handled it really well. I almost feel sorry for Netgate that they lashed out at someone so good at managing the technical (code), social (community), and political (bullshit) sides of a software project. Almost.
https://news.ycombinator.com/item?id=24430424
https://mail-index.netbsd.org/tech-net/2020/08/22/msg007842....
https://mail-index.NetBSD.org/current-users/2020/08/22/msg03...
https://mail-index.NetBSD.org/tech-kern/2020/08/23/msg026693...
No one has to be the bad guy here or end up hostile to open source.
This was made worse by the unfortunate timing -- the final release candidate is just 3 days away. Any other time, we would have gone slower, had more discussion, et cetera; unfortunately this turned into an emergency.
I think, we all should be way more cool headed. Email doesn't contain the tone and it is very easy to formulate something in a bad way when exhausted or upset. Some people use very strong words that considering their usual style aren't nearly as strong in reality.
In this case, the great question is, how did such a bad code (according to Jason) land in the code base at all? Don't other people look at it? Do the FreeBSD developers ignore concerns of other fellow developers until it blows up to beyond community communication circles?
Must have: in-kernel WireGuard for NetGate products.
Nice-to-have: a general-purpose FreeBSD kernel WireGuard.
The scope crept, and a piece of code that might have been fit for some purpose (whatever limitations NetGate has for its network stack, like "no jumbo frames", etc) was recast as fit for all purposes.
I guess Colin would have some insight about the process by which a ports-grade kernel module gets put on track for release in the kernel itself.
My take on this is that absolutely no part of the development process that led up to this is Jason's problem. It is not Jason's job to understand or assist or comply with NetGate's product development process; his concern seems to have been, justifiably, exclusively the proposed FreeBSD OS feature. I get the sense that a lot of the friction here comes from pfSense-types thinking that any part of pfSense is everyone's problem.
My question basically is: "How is it possible, the code of such quality was even seriously included in a branch of FreeBSD that would have been released if nobody would step in?" (That is if I understand the whole thing correctly and Jason's assessment is correct.)
This system isn't foolproof, but it generally works very well. There have been discussions about moving to a "mandatory review" model, but there is concern that this might overly slow down development in the context of a volunteer-run project.
It does everything my pfSense install did and then some. Eg DNS blocking and IP blocking are built-in instead of needing a pfblockerng-style plugin.
The only thing I've found worth complaining about is the accordion sidebar UI thing makes it hard to middle-click-open new browser tabs, because it's not obvious which entries in the sidebar are actual pages and which are fake hyperlinks that just expand the accordion submenu.
Basically, all the opensource claims don't amount to a hill of beans, because you cannot compile pfsense yourself, even for their hardware.
(I'm sure someone could come up with the link)
The firewall should be the ONE place where this would be critical. You have to run their binary.
I also think it phones home.
https://cgit.freebsd.org/src/commit/?id=74ae3f3e33b810248da1...
> Traditionally, network protocols are specified in a document of protocol behaviors. Then different organizations implement that specification. Then everybody interoperates and all goes well. In practice, it often doesn’t go well (see IPsec woes), but this at least has been the traditional way of doing this on the Internet, and in some ways it works.
> But that is not the approach taken by the WireGuard project. In contrast, WireGuard is both a protocol and a set of implementations, implemented with a particular set of security and safety techniques. That’s a radical departure from the traditional model, and one surely to raise some grumbles amongst graybeards. But I believe this is a necessary and beneficial quality for having the types of high assurance software that is needed for core Internet security infrastructure. When you use WireGuard, you’re not just using some protocol that is capable of producing packets that are legible by others. You’re also using an implementation that’s been designed to avoid security pitfalls, and that provides interfaces for using it that mitigate footguns. In that way, the WireGuard project is more expansive than a mere protocol project or a mere software project or a mere cryptography project or a mere specification project or a mere interface project. It combines all of those things into a single unified approach. (For this same reason, the original WireGuard paper [2] has been difficult for folks to categorize. Is this a systems paper? A networking paper? A crypto paper?)
> Because of that, I think this was an understandable predicament. After all, why shouldn’t a company be able to task a developer with writing some ring-0 WireGuard code in C? And why does it matter to me whether the code is garbage if it can at least produce protocol packets? The reason is that the WireGuard project’s mission is wider than that. We deeply care about code quality and implementation particulars.
> While we now have the FreeBSD code in a maintainable state, there are other projects too that could use some attention from us.
I get that WireGuard is his baby, but sweet jesus...