Freenginx: Core Nginx Developer Announces Fork of Popular Web Server
infoq.com
infoq.com
I actually don't understand why I am seeing arguments like this all the time. Rewrite takes time and effort. It may be OK for new projects. But for a mature project that has a quarter million lines of code and load-tested for cross-platform compatibility and security, it is way more than "just biting the bullet". (Not to mention that memory safety in Rust is ... still improvable. [1])
Rust is not a panacea. [2] When a corporate tries to seize control of a open source project, some one (or some entity?) has to do something, and rewriting in Rust by no means reclaims the freedom and openness of the project.
[1] https://news.ycombinator.com/item?id=39440808
[2] https://ariadne.space/2023/12/07/most-breaches-actually-begi...
So I think it's pretty clear that it's not a fork.
To add to the confusion, it seems like they copied the site with all the bug reports as-is, which obviously would only apply to a fork, not to a project that hasn't even got any code yet.
Yeah, they wanted to have some CVEs for a bunch of bugs in experimental code, the author of the fork didn't.
If not, I might agree that assigning CVEs are not helpful.
But if the code was included in one or more stable release then I think it should probably have CVEs assigned. Depending on how serious or real the problems were.
I don't know much about these nginx bugs, but it seems like they were found in real live production systems, not by researchers trying to bump their CVE credits. Also it seems like quite a few people in the world are using nginx for HTTP/3 and quic, so trying to say "it doesn't count because it's not on by default" seems like a bit of a stretch.
Should you be able to file CVEs against code posted in GitHub comments? StackOverflow Answers? Experimental branches on a maintainer's personal fork of the project?
The steps required to use HTTP/3 in the open source version are also outlined in the official documentation, and you can find numerous guides for using Nginx with HTTP/3, demonstrating that the feature has reached some level of public adoption.
This is in no way comparable to random snippets of code you find online, or in some experimental branch of a code repository that's not intended for public consumption. Filing a CVE for software that early adopters are likely to be using in production is completely justified, I'd certainly want to know about it.
[1] https://www.nginx.com/blog/nginx-plus-r30-released/
> Native support for QUIC+HTTP/3 – NGINX Plus now has official support for HTTP/3. [...]
> The QUIC+HTTP/3 support in NGINX Plus R30 is available as a single binary – unlike the experimental HTTP/3 support introduced in NGINX Plus R29, which had a separate binary for nginx quic. [...]
> Full HTTP/3 support is added. NGINX 1.25.0 mainline version introduced support for HTTP/3, and this support has been merged into NGINX Plus R30. The NGINX Plus R30 implementation has the following changes when compared to the experimental packages delivered in NGINX Plus R29 [...]
Nginx, the open source project, was safe. No CVE should be assigned to it.
NGINX PLUS, a separate and independent corporate product, was vulnerable. A CVE should be assigned specifically to it.
The issue is that the maintainers of the open source nginx project were paid by the company behind NGINX PLUS, and the company demanded the CVE be assigned to the open source part.
I think that's all that matters. How else do you propose that users are informed about vulnerabilities in their installation? That's what CVEs are for.
I'm pretty sure that setting arbitrary compile flags is enough to cause vulnerabilities in most software.
I personally ran nginx without this feature enabled because it was explicitly marked as experimental and potentially unsafe.
Having nine gazillion open tickets is helpful for no one.
Devs will have to wade through open tickets that are not meant to be worked on time over again.
Tickets that should be worked on will be lost in the sea of irrelevant tickets.
People affected by the issue will see the open ticket and think it means that it’s going to be fixed.
Is there ever a situation where a bug exists so far back in the history of the project that it doesn't make sense to fix? I'm asking this as an honest question and not a challenge. I can expect that some users still use what I'd consider to be legacy S/W but I wonder how far that goes back. If the developer(s) designate a version as EOL does it make sense to keep the bug open?
And OTOH, it conveys information if the developer marks a bug as "won't fix" rather than leaving the expectation that it might be addressed, someday.
F5 employee says code the was marked as "experimental" but the issue is that customers were running it in production:
https://news.ycombinator.com/item?id=39378523
https://news.ycombinator.com/item?id=39379984
It's worth going through that entire thread and Ctrl+F search userid "MZMegaZone" to get F5's rationale for assigning a CVE.
Why are some customers running experimental code in production?!? I can only assume they do it to solve a problem and take early advantage of a new feature that doesn't exist in the stable release.
So technically the CVE should be filed against NGINX Plus, not nginx ;)
I, just as lots of others, have had to terminate business relationships over "last straw" types of infractions and misbehavior. I've never regretted doing so, but cutting ties after trying to make a situation work is not something I've ever been eager to do.
I'm glad that I've never had to make that decision with something as well known as nginx.
Maybe there's a survivorship bias in that only the ones that go poorly people talk about?
JS has a different lifecycle though, I suppose. Everything gets rebuilt like 5-10 times more often than other on platforms.
Getting to the finish line matters more than using the absolute best fit so to speak.
Same goes for orgs that have a lot of JS talent but no Rust for example.
I've seen such efforts and they tend to be extremely complicated and not really doing much.
I don't assume rational agency but instead that many people make poor decisions.
The core language is okay enough, not my favorite but it works. It's the surrounding ecosystem that's a pain.
I would have to think really hard if someone asked me if I'd rather have all the infra scripts written in Perl or TS. TS is more likely to break, but easier to read and fix. The odds of Perl breaking are slim, but it is a fucking nightmare to fix. I'm pretty sure I'd rather have Perl than JS, though. JS without a type system is rough, to me at least.
Well... if you're pre-qualifying the hypothetical rewrite with "large", that already means we take as a given it will require significant time & effort. (Unless there's a sufficiently smart automatic transpiler tool like a future ChatGPT-v17.)
There have been some successful big rewrites. Google rewrote the web crawlers from Java to C++. They also rewrote the search results front page from Java to C++. Microsoft rewrote C# compiler from C++ to C#. Go rewrote the compiler from C to Go.
In any case, for curiosity... I ran the CLOC tool to count lines-of-code in the main NGINX repo ./src/* subdir. A project to convert ~165k of C to Rust is small enough to be doable by a small team. "Small" being 1 or 2 developers. (I'm not recommending they rewrite in Rust but just comparing the scope of the work to the resources.)
Semi-related footnote is Cloudflare replaced NGINX with in-house Pingora and that was written in Rust: https://hn.algolia.com/?q=pingora
-----------------------------------------------------------------------------------
Language files blank comment code
-----------------------------------------------------------------------------------
C 256 54041 6165 153818
C/C++ Header 134 4810 1077 10041
Perl 2 45 18 112
SKILL 3 24 0 99
Bourne Shell 1 30 5 78
make 1 9 0 21
C++ 1 9 3 19
Windows Resource File 1 3 2 1
-----------------------------------------------------------------------------------
SUM: 399 58971 7270 164189
-----------------------------------------------------------------------------------I guess you'd consider librsvg in the poorly bucket, whereas its maintainer considers it a success. I think there's a distinction to be had between "maintainer lead RIIR" & "rando RIIR"
https://lwn.net/Articles/771355 https://viruta.org/librsvg-rust-and-non-mainstream-architect...
It seems many programs are getting Rust replacements in one shape or another. If not being replaced it is, at the very least, being included in existing projects. The linux kernel now supports Rust which I believe, at this time, for Device Drivers. I also heared Rust compiler is included in the Windows Kernel.
Whether you/we like it or not, Rust is a serious contender! As a developer myself, would prefer to focus my energy on Odin or Zig but the reality is Rust is where job security is likely to be.. alongside languages like Java, Python, C#, etc.
Over the years, Job adverts have evolved from "must know OOP" to "must know design patterns" to "SOLID principles" and... now is likely to be "must focus on memory safety!"
For many, which Paul Graham referred as "the pointy haired boss" in his Revenge of the Nerd peice, is just cool buzz words of the time. It is just the current thing we need to do/use. Rust is going to gain the edge with these buzzwords.
This brings me back to projects like freenginx. The reality is new developers coming in are not focusing on C or C++. Academia is likely moving away from them in the coming years, in favour of Rust. The reality is new projects in C or C++ might not last when alternatives are being written in Rust. As it has "memory safety" it will likely be the default choice many look for, especially in a business environment.
As I have mentioned, I have been focusing my energy on Odin and (a bit of) Zig. However, the reality is there are not going to be jobs using them.. compared to Rust, C#, Python, etc. This means I will have to bite the bullet... and soon... learning Rust.
Have a look at:
https://github.com/nginx/nginx/blob/master/src/http/modules/...
It's got the whole checklist: nginx idiosyncratic module system, inline parsing, custom utf conversion, buffer preallocation and adjustments, linked lists, comments about side effects of custom allocator, and probably other things.
It's not easy to deal with source like that and any serious improvement to that area would effectively be a rewrite anyway.
Since anything doing work in nginx is a module anyway, it wouldn't even have to be a full rewrite in one go.
It might be a worthwhile experiment to take a common set of HTTP libraries in Rust and add Nginx-specific configuration parsing and some other bits together and see how far that kind of hackery would get. With a good BDFL and some community involvement, it would be a great way to get the ball rolling.
Amen to that.
I don't understand why more people don't highlight Rust's main weakness. The lack of strong stdlib.
No stdlib means you have two choices:
(a) Re-invent the wheel; or
(b) Use a crate.
Which means you open yourself up to supply chain attacks, unless you constantly audit the upstream crates, which, let's face it, most people won't, they probably won't even audit the original version they pulled in, let alone the updates.
Which means with something like an nginx re-write in Rust, which is a complex piece of software with many moving parts, you end up importing a whole ton of random crates. Half of which will probably become abandonware within a few years, because that's the way of the world with open-source, maintainers move on to bigger and better things.
There is a difference between rewrite in Rust and rewrite properly in Rust.
No doubt there are people out there doing solid open source work with Rust (e.g. the guys at tweede golf) but that takes time and effort, and in the case of tweede golf, the extra effort is financially sponsored, something that is not the case with most open source projects, especially the single-maintainer ones.
Or indeed clients.
In 2024 you shouldn't need to import third-party code to make a call to a REST endpoint and parse the JSON. Or the same with crypto.
All the stuff that's bread-and-butter in today's cloud-centric world, you shouldn't need to farm out to third-party code, it should be stdlib.
I'm sorry, but I'm having great difficulty following your line of argument.
Are you seriously suggesting that Go does not have any mature projects ?
If not, are you seriously suggesting that the mature Go projects in Go are encountering maintenance nightmares because of stdlib ?
Your line of argument is giving off a whiff of FUD, sadly.
A good middle ground here is to have official crates/pkgs managed by the core organization that can be upgraded separately.
I used to develop in Java in large organizations for many years and be responsible for deciding how we use the language, what libraries we use etc. I used to be pretty hard on developers who would choose third party libraries for anything the standard library already delivered. Even in cases where third party libraries were perhaps a bit better than the standard library. They'd have to offer a clear and significant advantage over the standard library over time to warrant consideration. Especially if we're talking about core functionality or something that would be expensive to replace later.
Every third party dependency you introduce is a potential problem because you have to track them. You have to care about their development and release practices, you have to track their license (which may change), the health of the project, you have to know how they respond to defects being discovered etc. You have to do this for every single piece of third party code if the code you work on is critical to your business.
My real-world experience is that third party code carries a much higher risk than whatever you depend on from a standard library and you can make mistakes that end up being extremely costly.
What on earth are you on about ?
An assertion was made that Go stdlib was somehow "a slippery slope though and causes maintinence nightmares for mature projects".
It is a rather bizarre assertion to make, especially in relation to Go.
I am therefore merely asking for evidence of that assertion. Because, frankly, it smells of FUD and you know it.
But instead you are the one looking to shut down the conversation by not answering the perfectly reasonable questions I posed.
That being said, a good reason for using Go and its standard library is that they have been very good in terms of keeping important things in good shape and they put a lot more care into smoothing upgrades than is usually the case for random libraries. That's much of the motivation for using standard libraries: they have to be maintained in a manner that doesn't blow up billions of lines of running code. Third party libraries tend to be a far more mixed bag.
As for crypto: I shudder to think what it was like to use SSL/TLS in the C/C++/Java world. Horrific implementations that are fiddly to work with so you won't be inclined to mess with them more than you absolutely need to in regimens that rarely, if ever, pushes you to improve security as algorithm preferences shift over time. We've made heavy use of the crypto parts of Go to build our own PKI solution for an IOT platform and it was a lot easier than in any language we've used before. By a good margin.
I think you inadvertently made the argument for using Go and its standard library :-)
(Of course, 10 years down the road, perhaps people have abandoned Go and it has become a wasteland and there's some other language with a better stdlib you should be using. But ask yourself this: how much of the code you have written in the past is still running unchanged? You'll have to evolve, develop and adapt over time no matter what so if Go were to fade away, it isn't going to happen overnight and you'll have plenty of time to move on)
I think you're misunderstanding OP's theoretical. If TLS is in the stdlib, that means its version is tied to the version of the language you're using. I.e. if you want to upgrade to a newer TLS, you need to upgrade your entire language version (compiler, interpreter, whatever).
So you can end up in a spot where you need to go from v1 of a language to v2 to get a modern TLS implementation. However if v2 of the language also contains changes to how the language handles threading, you now need to rewrite your app's threading to be compatible with v2 of the language, just so you can get a safe TLS implementation.
Basically because TLS is in the stdlib, TLS can force you to upgrade your language version.
If TLS was 3rd party, you could upgrade the library without upgrading your language version (presuming the TLS library supports your language version).
And if it does happen, it is likely a lot of other people will find themselves in the same boat, making a practical solution emerging more likely.
And note: the same thing could happen, and is more likely to happen, if you use a third party library. Some of the libraries I’ve used for years are no longer actively maintained, for instance.
Putting things like that in the stdlib makes them subject to the same API stability constraints as stuff like the threading API.
They tend to ossify and get abandoned when a 3rd party lib makes a faster, or better, or just more convenient API.
Go has a fair bit of this. I haven't touched the stdlib logger in forever. The HTTP server side is kinda meh, the client side is okay but not fantastic.
I am a fan of Go's experimental packages. I kind of wish things could live there permanently as "things the core devs believe are a good implementation, but don't want to bind stability guarantees to the language".
I would argue that Go is a fair comparison, Python is not.
Python is just a mess of a language, dependency hell etc.
> Go has a fair bit of this. I haven't touched the stdlib logger in forever.
You do realise Go has log/slog in stdlib now ? So structured logging is now in stdlib.
I do, it's just twice as slow as the old stdlib logger, which was already twice as slow as 3rd party loggers, and it makes like 8x as many allocs as zap. If I were going to change my logger, it would be to zerolog, not slog.
> Python is just a mess of a language, dependency hell etc.
We're talking about stdlib though, which are the only packages dependency hell doesn't apply to.
> I would argue that Go is a fair comparison, Python is not.
Go is the youngest of the major languages, making it one of the worst place to check if you want to see how packages age in the stdlib. Look at a language that's actually old if you want to see the kind of problems old apps face.
It also has a weird place, language-wise, because its developed by Google who has an unusual amount of influence over developer patterns. Other languages have to follow devs to where they are, Go can kind of lead the way via Google.
There are things I'd love to see in this space. For instance I'd love a proxy that explores the idea of being fully configurable via a gRPC API and which can delegate decisions to some other service (over gRPC).
(Doesn't the commercial Nginx have some of these features?)
The software was initially released 20 years ago. It was designed in an era where most http requests served static assets and html w/o js, and https was for special situations like banks and order forms. At that time multicore CPUs were just emerging, and servers still shipped with 1Gb ethernet. JSON was a quirky idea, php was king, and Gmail was shaking up web-dev with this funky new Ajax thing[1].
Point being, things have changed - these days there are a lot of situations where you wouldn't even consider nginx since it's not the right tool, there are situations where nginx is no longer the easiest component to use and/or deploy, there are situations where it's a giant hassle to maintain an nginx deployment compared to tools written for a "servers are cattle" world.
Put another way, the web was 13 years old and the internet was a little over 30, and both had really only gotten big over the previous decade. If we were talking aviation, it would be arguing to retrofit the best fighters from WW1 on the dawn of WW2 or asking why we can't just strap jet engines on a DC3.
nginx has been an amazing bit of software and a solid workhorse getting us here. But more and more people are looking at other solutions, because the problem to solve isn't the problem nginx was designed for. That's ok - the world changes all the time.
Software isn't done, we haven't even had computers for a hundred years. Heck we haven't even finished figuring out all the ways to use computers yet. I'm leary of any claim that we fully solved some aspect of it, and more so when the claim is applied to tech from the early days of that field.
[1] it wasn't new new, but this was the first time most people really started to grasp what you could do on the web with javascript.
For example, https://freenginx.org/en/security_advisories.html states "All nginx security issues should be reported to security-alert@freenginx.org." and then has a complete history of bugs from prior to the fork (and thus were never freenginx issues).
What exactly to display on the security advisories page on a brand new fork project with no vulnerabilities yet seems like it could be a particularly hairy problem. You want users to visit the page and see the schema of information they would get in a security advisory, so they are prepared to use the website if they need to. So you could display an example advisory if none are available. But at that point it would be easier to keep the old advisories, because they also help to inform users on why they should upgrade from pre-fork versions.
From the post it seems their intention are good, and towards better secured software, and against business interfering with security. But the part where they say "I no longer able to control which changes are made in nginx within F5" might be a good thing. Should a single person have that much control over a critical piece of infrastructure?
I'm not saying Russian people cannot make safe and secure software, just saying that people living in Russia have historically been targets of pressure.
* and to be honest, the USA and China do not have the cleanest of records either.
Everyday Russian developers are no more or less involved in their country’s empire-building than everyday American developers are in theirs (lest we forget about the 12(!) foreign countries in which US forces are currently invading). “Russian” is not synonymous with “suspect”; or, if indeed it is, perhaps “American” should be, too.
(yes caddy takes care of upgrading your headers, websockets, certificates, all of it)
never touching it again
good defaults matter; nginx configs were very concise and readable in 2005-2015 when I was using it heavily; http/3, ws, acme — perhaps new tech is yet to be incorporated properly (and maybe it never will)
However, Nginx gets over 20% more concurrent users per host... Hence the main reason people put up with its lack of features and silly HA design.
Forks can be healthy, but it usually quickly degrades into a competition for users.
Hope everyone has fun =)
You are correct in that many of the CVEs were not technically Apache itself, but rather supporting libraries and old mods.
Best of luck =)
Nginx is a stable, and reliable piece of software. It was before this, and will be after this. What happened is that someone who used to work on it now works on a new thing. This happens, people are not slaves to their previous projects. They can and should do new things when they wish to.
In time if the new thing becomes usefull in some way perhaps some people will choose to use it instead of nginx. Perhaps that day will never come. Both of those are fine.
What is certain is that nginx has a critical mass behind it that even if a core developer moves away to do something else it won’t shrivel up and die. If you are an nginx user today in a personal or professional capacity most likely you will be able to use it until the end of your life with no special worries.
I have no problem with you using Apache httpd, and I am happy that you are happy with your choice. On the other hand it does not feel usefull to portray this as some special reason why one should not have choosen nginx.