Cloudflare has replaced Nginx with in-house, Rust-written Pingora | Hacker News Reader

Cloudflare has replaced Nginx with in-house, Rust-written Pingora

phoronix.com

334 pointsAvlin67141 comments

Original post on Cloudflare blog -> https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...

Discussion (106 comments): https://news.ycombinator.com/item?id=32836661

Two days ago.

(I was going to share that link as well...)

The most important part if the blog entry is that they are building on top of async Tokio but not Hyper, so other companies can probably consider Tokio as production ready for large scale workloads.

Do you think there’s problems with Hyper?

I believe the lack of using Hyper was so they could be more flexible in what nonstandard parts of HTTP they could support, e.g. status codes into the 900s.

Hyper conforms to HTTP but a bit too much. Cloudflare has to proxy users who may send invalid HTTP like status code 666 or something.

And conforming to HTTP is a good thing when you are writing a server, where you want all your responses to be perfectly formed. But for a proxy that is mostly passing through other people's traffic unaltered, it is not so important.

You could almost argue it's counter productive to an extend. One of these invalid response codes might become valid in the future and then suddenly your proxy breaks valid communication attempts. One of the reasons TLS1.3 has to look like TLS1.2 in the handshake.

I don't know about HTTP, but TLS had explicit invariants. You could write a working TLS 1.2 proxy in say, 2010, and do it correctly, having of course no idea how TLS 1.3 will work, and it would have worked with TLS 1.3 as conceived over a year before publication, back when it admitted it was TLS 1.3 (0x03 0x04 ~= SSL 3.4)

But real world systems, most prominently "middleboxes" often sold as security devices, did not obey the invariants, they were actually bad implementations of TLS 1.2, but they worked and so they'd been able to proliferate. TLS 1.3 as eventually standardised needed to cope with that nonsense.

So, if there are HTTP invariants, then a proxy merely needs to get those correct and would in principle interoperate despite newer HTTP versions. With TLS 1.2 invariants talking to a TLS 1.3 client that meant a proper proxy would shrug and say "I can't speak TLS 1.3, you need to talk TLS 1.2" and that works fine. Whereas the middleboxes tended to go "OMG. A byte I didn't understand! We're under attack! Light the beacons, all men to the battlements!"

Isn't demo's also built on top of Tokio?

Yes

And hyper, for their native HTTP extension (for http2).

https://github.com/denoland/deno/blob/main/ext/http/lib.rs#L...

s/demo's/deno

In current state of affairs you can use async Rust as long as you use Tokio anyway, given that most crates aren't portable across async runtimes.

At cloudflare-scale, I think I would want a proxy that sits at least partially in the network hardware.

Ie. so that the bytes of a large image or video that is coming from an origin server and being sent to a client never has to pass through the system RAM or CPU.

I'd probably implement this as mods to the sendfile() API so that a certain number of bytes can be copied from one socket to another, with that request going all the way to the firmware of the network card which will do the actual work.

It probably needs to work with HTTP/3 UDP and encryption too - so decrypt the data from this socket and send it to that socket reencrypted with this other key and packetized to this HTTP/3 stream. The firmware would need some way to do aborts/timeouts and kick a partially complete request back to software too.

Is it complex...? Yes. But will the compute savings be worth it...? At cloudflare scale, I think the answer is yes.

We constantly think about the possibilities of using exotic hardware for acceleration. So far we've got a very, very long way with "commodity" hardware and the Linux kernel. One day, we'll probably do something exotic.

https://blog.cloudflare.com/cloudflares-gen-x-servers-for-an...

https://blog.cloudflare.com/tubular-fixing-the-socket-api-wi...

Just curious, is there a high cost overhead (aside from the cost of actual units) of e.g. custom silicon?

It’s just a question of “is it worth it?”. How much does this cool accelerator thing cost? How much does it save?

Why not using the same kind of solution as Netflix does where they offload TLS and the like in hardware?

https://www.nvidia.com/en-us/networking/ethernet/connectx-6-...

Pretty sure they analyse TLS handshakes for their Bot Management product and probably other things.

Have you tried out io_uring? My impression is that these Rust libs end up making different syscalls that are less great.

Tokio supports io_uring (https://github.com/tokio-rs/tokio-uring), so perhaps when it's mature and battle-tested, it'd be easier to transition to it if Cloudflare aren't using it already.

Don't assume io_uring is an automatic win.

Last time I benchmarked io_uring, it was good enough, but I was able to get measurably faster from syscalls and a well-crafted userspace thread pool for storage to NVMe.

(Note: the I/O thread pool I wrote came out noticably faster than fio, the benchmark tool, so don't assume fio results are the best possible.)

For that reason, I have a small application which can be configured for either io_uring or the thread pool, and the thread pool is preferred for performance.

I don't think the hardware need be very exotic - most network hardware already has firmware capable of various types of acceleration, and all you would have to do is partner with the network card vendor and have a joint team write firmware that has all the features you need.

With shared source, the network card vendor can even repackage some of the types of acceleration you write for their other clients - and I bet they're already looking into acceleration of HTTP/3.

It's a win-win - because cloudflare gets massive compute savings, and the network card vendor knows they've built quite a high moat to buying someone else's hardware.

Cloudflare has generally been very conservative about its tech stack (eg kernel TCP, well-known OSS solutions and abstractions), which seems to work fine for them. They are certainly leaving performance on the table, but they also don't have to pay to support exotic solutions.

Other firms in the same space do things that are a lot more exotic, and it is worth it for them.

If someone from Cloudflare is reading... please don't do this. I'm asking for two reasons:

1. Cloudflare has been great at rapidly iterating (compare the timeline between HTTP/3 support in Cloudflare vs Cloudfront... let alone AWS's ALB that still doesn't support it). Introducing hardware accelerators would surely hinder those efforts (ouch, we need Y to do this that the hardware can't do, and the vendor says it'll take 1 year to have new production-ready cards).

2. Cloudflare has been a good upsteram contributor for the projects they depend on (the kernel, the rust language, etc.). Partnering with a hardware vendor inevitably means closed source, deviations from upstream and ultimately a much larger hurdle to be a good citizen to the open-source community.

As you can see, the reasons are entirely selfish. I'd understand it if you do it anyways because the numbers make sense for your company. In the meantime... thanks for holding out until you really can't justify it anymore!

Those are actually reasons I'm hesitant. We want to be able to support protocols etc. very quickly and can't wait for hardware to catch up. As I said elsewhere the combination of CPUs and the Linux kernel has worked pretty well for us.

> Partnering with a hardware vendor inevitably means closed source, deviations from upstream and ultimately a much larger hurdle to be a good citizen to the open-source community.

No, it absolutely doesn't "automatically" mean that. You can totally request some major hardware vendors to fully upstream hardware offload capabilities in their Linux drivers.

Main takeaway: Unless you are operating at Cloudflare scale nginx is probably fine.

And considering there was no real need to move away apart from the difficulty getting new features I would say even if you are operating at Cloudflare scale Nginx is still fine! (Although they did save hardware resources by moving to their own proxy and increasing cache hit rate - but seemingly none of these were business killing issues.)

Did anybody have any doubt that nginx is fine?

What is newsworthy is that there's a new server that's validated as fine.

Weird that they didn't use HAProxy in the first place if they are (apparently) not even using it to serve files from filesystem

We had a great deal of experience with NGINX and the original architecture for Cloudflare was based on NGINX. When we came to move away from it we wanted to fully own our destiny (as we had done years before when writing our own DNS server) and so moving to HAProxy would not have made sense.

They were wondering why the original architecture was using NGINX over HAProxy, not why HAProxy wasn't chosen as an NGINX replacement.

I think because Lee Holloway (the technical founder of Cloudflare) was used to use NGINX and so he used it for the original architecture. It's also the case that Pingora replaced part of something that handled both connections to origin servers and reading from cache. So the comment about "not serving files" isn't 100% correct as the NGINX instance was serving files as part of its work.

Well, it does offer more varied stuff from the get go so I can see why someone wouldn't want to limit their options, HAProxy was and is purely a proxy while NGINX is a bit of everything.

SPOE/SPOA added a bit of programmability to HAProxy but it is still basically only messing around with headers and acting upon that, nothing to do with content.

Back in the day, Cloudflare's WAF was based on OpenResty, so the high-performance Lua-programmability at the edge (which is noted in this blog entry) was probably a factor. Quick research shows HAProxy added Lua support in 2015, which is a bit later than their use of OpenResty.

Yeah, I know, I wrote that WAF.

This exchange is almost as enjoyable as that "famous" thread about someone's math prowess (anyone have that link?)

https://news.ycombinator.com/item?id=35079

Haha, thank you.

Haha, I missed that you had an intermediate comment -- thanks for all you+Cloudflare did for LuaJIT too :)

LuaJIT is a cool thing. Helped us a lot. But not everything remains the right thing forever.

nginx certainly did serve cached files from the filesystem.

Lee actually looked at HAProxy when we were building the first prototype of Cloudflare but found he could get better performance and extensibility from NGINX. Things obviously evolved, but we considered HAProxy early on and decided to go another way.

Was curious if the Cloudfare team evaluated other languages for this effort (I thought they used Golang quite a bit at the company).

We use Go and Rust and C++ and all sorts of things. I wasn't involved in selecting Rust for this project but in general we've seen Rust be a very good fit for things that might long ago have been written in C.

https://blog.cloudflare.com/boringtun-userspace-wireguard-ru...

https://blog.cloudflare.com/building-cloudflare-images-in-ru...

https://blog.cloudflare.com/enjoy-a-slice-of-quic-and-rust/

so anyone knows Tokio is production ready ?

I think io_uring support [0] still is not production ready.

[0]: https://github.com/tokio-rs/tokio-uring#project-status

Rust in the title? Straight to the front page.

Jokes aside that's some serious performance boost. Wonder what it'll perform like in everyone else's prod environment.

It is not just the performance boost, but the fact they can support any non-rfc compliant bullshit there is on the web they feel like is necessary.

I wonder if as part of the rollout, they selectively routed non-rfc compliant connections to nginx and get 90% of the way there with their changes while they added more support

Wouldn't "switches to" have been a better word instead of "ditches" which implies they are getting rid of it? Considering Nginx is an old and reliable FOSS software...

What precedent would it set for developers when 10 years down the line, a new shiny tool would just "ditch" the result of their hard work?

source. very interesting

We are going to open source it.

This is an exciting prospect. Some financial exchanges are currently proxying their order entry and market data through Cloudflare, so it's useful for their customers to understand how the proxy works.

care to elaborate?

I've worked with essentially every exchange in the world and I've never seen http anywhere near order entry or market data

Binance does a modest 50 billion dollars of daily volume.

Good :)

Btw, I’d like cf to have a feature that will prevent anyone (including cf) from viewing my traffic. Is this in plans? I want the protection of cf but don’t want to sacrifice privacy of my customers data..

https://www.cloudflare.com/products/cloudflare-spectrum/

Thanks! The missing part is the WAF… unfortunately I couldn’t find anything for on premises installation at a pro cf account price level or open source.

Yeah. We don't provide an on-premise solution. It's all cloud-based. If you want us to do inspection we have to terminate TLS. We can do DDoS protection without termination but WAF requires inspection of the payload.

If your concern is where descryption happens then Regional Services might be a solution: https://www.cloudflare.com/data-localization/

"Also considered safer"?

The article is interesting and the solution great but saying it is safer because Rust is misleading. I wrote a Rust milter for Postfix incorrectly and although it didn't crash as such, it completely didn't work because data was coming into it from outside. i.e. it is still possible to write broken code in Rust.

I've personally never considered nginx unsafe either but tbh, I am not exactly an nginx guru on the cutting edge of performance or load.

> The article is interesting and the solution great but saying it is safer because Rust is misleading.

Using Rust (without unsafe) precludes memory management bugs being turned into exploits. That closes an entire category of attack.

Sure, “safety” can be defined along different axes, but Rust handles one entirely. How does that not make it safer than using a memory unsafe language?

> How does that not make it safer than using a memory unsafe language?

It was never compared to using a memory unsafe language. It was compared to a wildly used and battle-tested open source software.

Nginx is written in C which is probably where the comparison is coming from.

Cloudflare uses a custom fork of nginx, with custom extensions, Lua FFI, and improved HTTP/2 and caching modules. So it is a comparison to development of in-house C.

People die wearing seatbelts. They are still safer than not wearing seatbelts.

They've said "safer", not "100% foolproof in every circumstance, no exceptions".

Has anyone ever claimed that if it compiled and ran in rust that it was logically correct? I've never seen anyone claim that and would call anyone who did an idiot. People do say it will tend to have fewer memory and programming errors out of the chute than c or c++ though given similarly experienced programmer because the compiler is much stricter.

Reply on news.ycombinator.com