Writing my network protocol in Rust | Hacker News Reader

Writing my network protocol in Rust

ayende.com

87 pointsGordonS21 comments

It's interesting to note that the author's better experience with the language wasn't the result of any one improvement, but rather lots of little improvements, such as error messages, that overall added up to a better quality-of-developer-life.

I think Rust development is going to continue to be less focused on big features and more on small iterative improvements. Rust doesn't need silver bullets; it needs lots of lead bullets.

From personal experience Rust has the lowest “Google search to error message” ratio (if that’s a metric) of any programming tool I’ve used. It’s genuinely a delight to use after having to deal with error messages from various JavaScript tools, Python packages, and C/C++ build tools.

That all depends on how deep into macro territory you get. My experience has been that serde-derive and diesel.rs give absolutely arcane error messages. There are a lot of things that rust at least tries to get right, and that helps a lot.

That said, even with all the nagging faults I find that rust development feels more productive than lots of other languages. A large part of that is the tools, sure, but the other major part is the rust community itself.

It sounds like I probably haven’t gone deep enough into macro territory then.

Still, when having to do things with JS/Python/C++ I’ve wasted many afternoons down the rabbit hole of trying to glean information from Github Issues, stackoverflow answers, Twitter threads and obscure forum threads — usually to find the problem (and fix) I ran into was so trivial it could have been a single line in an error message. (I feel like a lot of JS tools - particularly Webpack, Jest and Babel - have regressed a fair bit recently in this regard. Python just fails silently (or bizarrely) too often and C/C++ compiler error messages are often less readable than the code itself.)

IIRC rust was designed from the get go to be user friendly. That's why you have things like intelligible error messages and rls. The rust community, as well, plays a huge part in how easy it can be to track down answers. But try debugging a nom parser sometime. Your eyes will bleed.

OTOH the contrast to Javascript is pretty stark. As much as I've got plenty of complaints about the language itself, the community really a detriment. Things like Electron and Node actively shun cross-platform compatibility. Tools like webpack tend to attract cargo cultists. It seems there's a lot of wizardry and hair pulling involved anytime I touch JS.

C/C++ though has gotten a lot better with the advent of clang.

Diesel.rs does give very scary looking errors but I found they are mostly due to the entire record structure being there as a (part of) type signature - after some time I started to get a “feeling” for what is wrong.

Also, over time, as I gained experience with the language, I found myself taking repeating code fragments and creating higher-order macros - both for Diesel.rs stuff. That reduces the repetitive code and errors. Macros in Rust are a great way to reduce a complexity where the functions can’t.

Source: during the past 1.5 years period in a couple of 2-3 month spurts I rewrote a C#/ASP.Net homegrown NMS dealing with provisioning and operation of a large-ish (about 15K people) event network consisting ~500 devices, all in all probably about 50KLoC of Rust code doing FTP/TFTP/SNMP and ssh/telnet interactions and this year the frontend.

Now (over the new year holidays) I am using the learnings from that to build a stateless-at-server ASP.Net-like framework using Iron, Diesel.rs, rust-Mustache and iron-sessionstorage on the server side and HyperHTML on the client side. I hope to open source it (the framework) at some point, when I get it to Rails-like ease of use.

Some additional impressions: I feel I am about half as productive with Rust initially compared to C# for smaller changes, but the final result is much more solid, requiring practically zero debugging. And as I build the abstractions later it becomes easier and easier.

Also it is a joy to get a consistent and easily traceable memory footprint compared to mono and about 10x the performance (in debug build) compared to the mono version. And the benchmarks show I have about 5x-7x more if I run it in release mode. So - a very happy Rust user here.

Honestly I don't think Diesel error messages are too bad, they're just extremely verbose to a fault. It'd be nice if it could filter out a lot of the duplicate information, or maybe just print spans. Usually when I get a Diesel error message it's of the form: "got <this huge tuple of types>, expected <this huge tuple of types with one of them transposed>." If diesel (or the compiler) could just put a span under the element(s) of a generic tuple that don't match it'd help immensely w/ the vast majority of errors I see from that crate.

What caught me off guard were the error messages when not enabling serde functionality for optional dependent crates (e.g. chrono). Nom, which I love, has been a matter of interpreting the error messages with a divining rod though.

Emphasizing which part of the tuple didn't match up would be a big win, but IMO it should be pulled out and mentioned first. Highlighting could get lost in the noise (moreso without all the ANSI escape codes if you're redirecting output or whatnot).

+1 on needing to specify “serde” as a feature for chrono. Another one was needing to enable “huge-tables” for diesel itself - some of my tables from the “legacy” system have many fields.

Not just chrono but bigdecimal too (diesel really needs some more explicit documentation IMO).

> Rust doesn't need silver bullets; it needs lots of lead bullets.

It needs steel bullets. Lead only oxidizes at high temperatures.

OK, gotta jump in here and ask the dumb question. What's the deal with this thing:

    lazy_static! {
        static ref msg_break : TwoWaySearcher<'static> = {
            TwoWaySearcher::new("\r\n\r\n".as_bytes())
        };
    }

AFAICT all the code is doing is parsing the input on a double newline (e.g. the equivalent of strstr(input, "\r\n")). And this five-line expression is defining the equivalent of the "\r\n" literal.

What is it that requires that mess? I mean, I get that sometimes typesafety requires some loss of concision, but...

The lazy_static macro (provided by a third party crate) allows you to create static (global, basically singleton pattern) variables that get initialized on first use and then are being reused over and over again.

The "lazy ref" is part of that macro's syntax. "msg_break" is the name of the static variable. Then, in Rust, all static things must have type annotations. In this case, it's the "TwoWaySearcher<'static>". The "TwoWaySearcher::new("\r\n\r\n".as_bytes())" is the initializer code that's being only called once. The as_bytes function is solely a type casting operation to convert from a str for which the typesystem guarantees that it's utf-8 formatted to a byte slice that just represents arbitrary data.

Actually I sort of knew all that (though it was the first time I'd seen lazy_static! and had to infer it from context).

The question was more about "why"? Again in C the equivalent is literally a string literal. There's really no easier way to express "double newline" in Rust than this thing which requires that the reader parse a "lazy static", a parametrized type, an explicit type conversion and a library utility? Why not?

This thing is so bulky because the API that TwoWaySearcher has is so bulky. You could also have a very simple API that just takes a string literal. It's less a problem of Rust and more of that TwoWaySearcher struct.

>There's really no easier way to express "double newline" in Rust than this thing which requires that the reader parse a "lazy static", a parametrized type, an explicit type conversion and a library utility? Why not?

The code doesn't express 'double newline'. That would be just the "\r\n\r\n".as_bytes()

The code lazily initializes an TwoWaySearcher to parse such double newlines.

Also note that b”” is nicer than .as_bytes()

One reason why someone might choose to do that is performance: it ensures that any precomputation required by the two-way algorithm is cached and only done once. After the first search, later ones can jump straight into the main search loop, rather than having to redo any set-up for the fixed needle "\r\n\r\n" (for two-way searcher, it's O(size of needle)).

If that setup-performance doesn't matter one can use https://crates.io/crates/twoway to write:

  twoway::find_bytes(&buffer[to_scan..], b"\r\n\r\n")

at the place where the search happens and skip all the caching effort.

Is not Ayende also a big expert in .NET. I am curious about his current experience with C# considering the huge improvements regarding memory management in the 7.x series of C#.

I'm still using C# (CoreCLR) for most of my day to day stuff. This posts series is part of my ongoing efforts to improve my knowledge and skills.

I've talked extensively about performance in CoreCLR and how to approach it. Several records talks are found here:

https://ayende.com/blog/tags/performance

I was curious especially on the network protocol stack considering that the ASP.NET team pushed the CoreCLR and language teams a lot to achieve a good performance with nice syntax and maintainability.

Thanks Ayende for all your content and contributions.

Reply on news.ycombinator.com