Rwasa – A high-performance web server in x86_64 assembly

> My zlib implementation for example is consistently 25% faster than the reference version, despite me simply "hand compiling" it straight from the C source.

Have you compared it to Intel's optimized implementation of zlib, at https://github.com/jtkukunas/zlib/ ?

If you can improve on that implementation, please consider submitting a patch.

It's the same in the high assurance systems field albeit with different goals. We're concerned with optimization-induced failures, subversion, complexity, and so on. Mainstream languages and their compilers... doesn't exactly help on this. I've been out of it a while, evangelizing & informing mainly these days. Yet, your post brings back memories.

My strategy, leveraging the Write Great Code book, was to map language constructs onto assembler via macro-assembler or languages such as LISP with good metaprogramming. Then, I hand compiled the code in other's projects with my macro's. So, we took a similar approach there although my goal was to show a correspondence argument between source & asm. Meta tools turned that into full program for assembling and linking.

One wild idea I had for portability was doing optimized routines of a safe HLL in a bytecode like LLVM. That knocks out most of the uncertainty of above layers that limit optimizations the most. Then, the simple optimizations that machines are good at can be performed from there along with generation of assembler. Close to portability of C and efficiency of hand-written assembler with inline available.

For instance, code zlib functions in pretty optimal LLVM and let toolchain do the rest on full-optimization. Think yours will be 25% faster, 10%, similar? And I'm talking what you can code quickly rather than spend 30min-1hr optimizing by hand. Just curious to hear your thoughts as you have way more experience in that stuff.

> Contrary to popular belief, x86_64 assembler isn't really that bad to deal with.

I think most beliefs regarding x86_64 asembly is largely "guilt by association" with i386... It's amazing the difference just from making use of the larger register set.

> My zlib implementation for example is consistently 25% faster than the reference version, despite me simply "hand compiling" it straight from the C source.

Yeah, but that is a very CPU bound processing pipeline. You would expect that to maximize the impact of any inefficiencies in the compiler.

That you can hand tune for better performance is conceivable. That you can get a 2x win over some pretty tuned code suggests that there is something larger at work than simply tuning lots of little things.

Would you care to give an example? The only thing I can think of is the typeless-ness.

This makes me think a direction worth pursuing would be to save the whole-program-analysis from the last compile pass for the new compile pass, so you can avoid re-analysing things. So you'd spend maybe 20 minutes on the first compile, then edit one .cpp file and the next compile takes 10 seconds.

Several more that come to mind:

* "unusual" control-flow: return several levels up from a function without requiring extensive elaborate exception-handling mechanisms, coroutines, and techniques similar to continuation-passing-style. After all, functions and procedures are just an artificial construct imposed by HLLs.

* Easily return multiple values from a function by using several registers, even normally inaccessible ones like EFLAGS (very useful for booleans.)

* Generating self-modifying-code, like a simple JIT compiler, is straightforward to do. Works especially well for tight loops that have several variants of their bodies.

Compilers are still bound by conventions and the features the HLL exposes. Asm is only limited by what the CPU can do (and what the programmer can come up with.) I admit that, while I do prefer using something like C for much of the "mundane" code I write, it's quite frustrating in those situations where I can think of a very elegant way to do something that either can't be expressed in C without some extreme compiler-fighting, or is completely impossible because of how it generates code and what the language allows.

Which comparably featured Haskell servers beat nginx? Curious.

No. Using something that truncates and rotates the log files is pretty common.

People generally use logrotate. And limit the amount they keep.

Yes - at least, from a sample size of 1 (me) I like the old cronolog way of doing this, and it means I don't get the rotate-a-day-later-to-avoid-trailing-lines-in-compressed-logfiles issue.

Depending on what you read [1] cronolog is a decent way of doing things (e.g. unlike logrotate it doesn't require a reload of web server). Without exporting to something like ELK, it also gives you a decent audit/accountancy trail. Limiting logs is not something I can really do in production (think six-years-plus of mandated data retention).

[1] https://startpage.com/do/search?q=cronolog+vs+logrotate

The generally accepted logging convention is that you work with how logrotate works (listen to appropriate signals for freeing up the file to let logrotate truncate it and so forth). If anyone needs to setup anything other than that default, it is typically better to target customizing logrotate and unify your endpoint logging there rather than to keep modifying each application's configuration. This is about separation of concerns for me as a sysadmin v. getting away from operations decisions. If your project is widely accepted you'll want to limit the amount of code you maintain on stuff that's not in your domain. Heck, it's really the Unix Way. If you were writing against Windows, I'd be using the event logger for application events with my own text based logs and let users configure what I emit to the main event logger.

An example of how people let logrotate do most of the lifting is in the Chef server codebase.

Thanks for sharing! Is there a repository I can subscribe to somewhere? I'd like to watch this project's development.

I also want to add I love the documentation.

I almost downvoted you by accident. Thankfully when I reloaded the arrows were still there. I reply think this "arrows to small" problem needs to be fixed. Although the refrain seems to be that it is not a problem: the community would vote such accidental downvotes back up

I'd be happy to give some more detailed thoughts. I write a lot of assembly myself [1], so hopefully that lends me some credibility.

I haven't read much of your TLS implementation, but I'm not a security researcher and I don't think I'm qualified to give an opinion on your particular implementation. However, there are some points to be made here. First of all, almost no one ships their own crypto for a good reason. To trust that something is secure, you need to have lots of people working on it and lots of projects invested in it (something that OpenSSL and such have). "Many eyes makes bugs shallow" is the common phrase, but it holds truer when large companies with highly skilled engineers are putting their secrets and their customer's secrets on the line. Your implementation has the unfortunate problem of being written in assembly. While I don't think there's anything inherently wrong with assembly, many people don't share the same opinion. People will be reluctant to contribute to it (how do people contribute, anyway?). Also, it is easier to make mistakes in assembly, and I would be surprised if there weren't several mistakes in your TLS implementation and in the rest of the server. This doesn't reflect poorly on you as a developer, but is instead a consequence of choosing assembly.

Assembly also inherits a lot of the common security problems C has (like buffer overflows), but makes them harder to identify. I would feel very uncomfortable exposing anything written in assembly to the public net, and doubly so if it used an unproven TLS stack written in the same. Other projects avoid the problem of untested crypto by using tested crypto from an external module like OpenSSL.

[1] https://github.com/KnightOS/kernel

Probably the amount of eyes that read it. This stuff is insanely easy to get wrong in very subtle ways and reimplementing the stack always carries a risk.

This is why people usually prefer to stick to the "regular" libraries even if they're known to be slower.

From the documentation:

> BREACH/TIME/etc > > Both the BREACH and TIME attacks rely on measuring the size of compressed response bodies. Since rwasa supports dynamic content compression by default, the HeavyThing library's default setting for webserver_breach_mitigation is enabled and set to 48 bytes. For each rwasa response when TLS and gzip is active, this setting adds an X-NB header that contains a random 0..48 bytes that is hex-encoded to each response header. While this doesn't render response sizing attacks completely useless, it makes a would-be attacker's job much more difficult due to the highly variable response lengths.

It's my understanding that random padding doesn't in fact make the attacker's job "much more" difficult. Only a little more, or not at all?

Could you comment on how integrated the TLS stack is with the webserver? Normally I'd think that using some kind of dedicated SSL terminating proxy, either a new version of HAproxy -- or stunnel/stud or similar -- would make more sense than deploying a new TLS stack that hasn't been through any outside review?

That said, as mentioned by others here - openssl is clearly not a great example of a secure/good TLS implementation. I'm not sure there are any (yet). Hopefully libressl will become one. Personally I'd like to see a minimal library that combined a couple of AES/ECC primitives and implemented TLS 1.2+ only (No SSL), with a sane and clean API on top.

Something along the lines of NaCl but with a goal to support a subset of standard TLS with forward secrecy (and explicitly throw old clients under the bus, Android 2x be dammed).

I'll add another slight spin, which is that I'd never run this in production, ever, even if I actually personally trusted you a great deal, because I can't possibly audit this sort of code base. By extension, for the same reasons I can't audit the source very well, neither can anyone else. (i.e., no, I do not personally audit everything I run but I reasonably expect that because it is possible others have. "Many eyes make bugs shallow" may be oversold but it is not simply false.) The only practical way I know to audit this sort of code base at this time is naked-eye inspection, and I don't trust that.

I say "I know" because there may be something out there. I know there exists tools for source code analysis that deal directly with assembler, because I see theses around writing them. I don't know where to get one, though, or how to use it, or how much to trust it. There's a lot more that deal with "C" or "Java" rather than raw assembler, so I can fire several tools, both commercial and open source, at the problem.

And all that said I'm still extremely strongly in the "STOP WRITING C CODE AND PUTTING IT ON THE INTERNET DAMMIT" side, even with that support. Without even that support, I frankly don't care if it's 100 times faster than nginx. nginx is already maxing out my risk tolerance as it is, and I've begun a long, slow program to get it out of my stack too.

I want to emphasize how this is explicitly (but also unapologetically) non-specific, none of this is personal, none of this is directly critical of your code (because if you've gotten the impression I haven't even glanced at it, that is correct), and in particular, please by all means do whatever you like with your spare time. The "problem" here isn't that you have somehow failed to leap my bar, the problem is that the bar is impractically high for code written in raw assembler. I suppose you could provide a math proof but I'd almost argue in that case the server becomes implemented in said proof language rather than assembler anymore.