Hacker News Reader

Comment by buro9 | Hacker News Reader

It should be noted that this implementation has been in production since last Summer and is powering the Firewall Rules ( https://blog.cloudflare.com/how-we-made-firewall-rules/ ) engine today.

This was one of the first major components to use Rust at our edge and we've been really happy with it.

The flexibility this approach (to matching traffic) has given us... whilst we also get the speed, memory safety... is just great. The speed at which we're iterating on the firewall and other systems that use this is a joy to behold. A lot of that speed derives from the confidence in this component.

We're also really pleased that whilst we have several proposals for optimisations, none have yet needed to be prioritised as the performance is great.

View on HN

> . The speed at which we're iterating on the firewall and other systems that use this is a joy to behold.

How much of this do you attribute directly to Rust versus other factors like seniority of engineering talent, clear communicated requirements, organizational empowerment, etc? If you swapped out Rust for C, what would be the impact on velocity? Is it a 1-4x multiplier, or larger?

Engineering resource is finite, the less time spent on working on the engine the more time we can spend working on new features and other systems.

Before making this in Rust we experimented with a Go and also a Lua implementation as we investigated the approach. The Rust code took the same time to initially produce and was production ready very early and has been rock solid and required virtually no maintenance since it was put into production.

That frees up that engineer to work on other things, whilst also reducing ongoing maintenance that was anticipated and so those engineers are also working on other stuff.

I'd attribute a chunk of that to Rust... though it helps if you have great engineers who are familiar with the concepts of parsers, etc working on these things. The same may not be true if someone new to Rust was also new to the concepts needed.

What were the differences between the Go and Rust versions that led you to choose the Rust version in the end?

Given that I wrote the Go version and RReverser wrote the Rust version, I'm going to go with: A better engineer wrote the Rust version.

But actually I was excited by Rust too. The memory safety, performance (the Rust was faster) and the degree of control over how we could present the FFI to the languages we needed to integrate with, in addition to how readable the Rust was by comparison to the Go code (readable, but so much of it), and then avoiding the GC... the Rust implementation was a convincing winner.

In terms of readability, how much is it due to Rust having ADTs and pattern matching over them?

You can kind of "fake" ADTs in Go with interface{} and switches, but it isn't all that nice.

Rust on the other hand with first-class ADT and pattern matching makes it a pleasure! You also have all the nice advantages of the compiler checking that all branches have been satisfied which catches a lot of bugs at compile time.

Go has it's place, and works great in those places, but I don't think this is one of them. I'm pleased we chose Rust and this feels like a great use for it!

It presumably also helps that the Rust version is the third iteration of the code (given that you mention earlier implementations in Go and Lua).

All of these were written by different people, and languages are quite different as well, so not sure that particular aspect played a role.

As I compare Rust and Go, the one thing that keeps me indecisive is that the Rust compiler is slower. Was that ever an issue for you?

It's not a big issue when you're working on the code continuously, thanks to the incremental compilation, but in general I'd always take slower compilation with more analysis (== lower risk of crashes and faster runtime) over faster compilation any time.

Depends, I work on large C++ code base, compile times are really hurting productivity.

You may try precompiled headers

It's important to keep in mind that there is always going to be a bottleneck on productivity. Sometimes that bottleneck is compilation speed, sometimes it's the language allowing unbounded growth in complexity, sometimes it's ecosystem immaturity, sometimes it's fixing bugs another language would have avoided, and sometimes it's a language's lack of expressivity. But there's always something.

It's worth keeping in mind that in some other language or with some other library or using some other toolkit, you'll still wind up with a bottleneck. It'll just be a different one (and maybe a worse one).

Does C++ have a "lower risk of crashes" as GP said though?

When I worked on a large C++/Qt code base compile times were awful. We ended up setting up distcc and that helped immensely. Worth looking into.

Right now Rust compile times are still slower than C++, when doing "make all", as everything is always compiled from scratch and crate binaries aren't shared across projects.

Delphi/Ada/D/.NET Native like compile times are still on the horizon.

Many years ago, I switched from working on a C++ project to one using Java. I was amazed how fast the Java compiled!

The downside: I used to use the "waiting for the compiler" time to think about what I'd written, consider alternatives, think about refactoring, etc.

Suddenly I had to block out time for that :-)

My "gentle" java introduction came by using GWT (java -> javascript). Also we had just one module, so it took quite a long time tom compile :)

At least most of the unit tests were run under java - e.g. the "business" (ui) logic.

In seriousness, maybe part of the power of C++ is how it forces the programmer to slow down, be more methodical and concentrate more. A language that is faster and looser, could also burn out the programmer faster and encourage shallower thought patterns.

I know that when I am trying to find a bug (any bug, it could be logical or data) that I am slowing down a simulation of the interaction of components in my head. This helps one understand all of the abstractions and evolutions of processes within the program. Maybe that very act is the power of C++.

Engineering resource is finite

Yes. So if one is going to do anything (optimize, write a new server, whatever) one should focus effort where it gets the most bang for the buck.

Before making this in Rust we experimented with a Go and also a Lua implementation as we investigated the approach.

This is excellent! More companies need to be willing to spike & experiment and change directions on effort up front. In general, the sooner you know, the more money you save.

Amen! But I know what my company would say. "Training devs and SRE on a new language is too expensive. And there are too few Rust devs in the job market, and the ones there are are too expensive to hire."

Cloudflare still seems to operate pretty similar to a startup despite now being huge. These days, it seems like a lot more companies that begin as startups are preserving more of that experimental, innovative culture to their later stages. Y Combinator, and others, definitely deserve credit for positively influencing the industry.

A few thoughts:

1. Cloudflare has an unusual structure in that we have a three engineering groups: the core engineering group that builds things that the product management team specify, a totally separate disruption group that works on riskier bets, and a crypto research group.

2. I strongly believe that letting people work in languages they love has a huge advantage. Engineers want to learn new skills (and those that don't we don't want to hire) and so letting them do that means they are happier and do better work.

3. Small teams do more than large ones. The teams that work on Cloudflare products are very small and agile.

On the other hand, I have experienced a situation where a fairly young group experienced tech burnout from having to learn too much at once. It's a really tricky balance. I was definitely on the side of "We're programmers! Learning programming languages is like drinking water for us" originally, but now I step back a bit every time we think of introducing new tech. Still, spiking a solution in a couple of different languages seems awesome and I can think of a few projects I've worked on that would definitely have benefited from that.

> Before making this in Rust we experimented with a Go and also a Lua implementation

Lua or LuaJIT?

was this project inspired by the Usenix ;login: article Safe Parsers in Rust Changing the World Step by Step ?

it seems the philosophy behind it is very similar

https://www.usenix.org/publications/login/fall2017

Never heard of it, but will check out, thanks.

after re-reading the article I think the similarity is nom. It probably ends there since the usenix thing is quite old in comparison.

After looking into it, I think I actually did come across it long time ago. As you've noted, the approaches are different, but it's indeed inspiring in the sense of moving to safer languages for parsing in general.

Reply on news.ycombinator.com