Writing high-performance servers in modern C++ | Hacker News Reader

Writing high-performance servers in modern C++

medium.com

126 pointsadamnemecek73 comments

A lot of people state that Python/PHP/Ruby/Javascript on the dynamic language side and Java and other JVM languages on the statically typed side and much quicker to develop than C++, but being a very experienced Java programmer and a decent Javascript programmer I find that I can program just as fast with C++ as long as you use the right choices of libraries and use C++11 or greater. With libraries like Boost, fbfolly, wangle, etc. I find it can be just as quick to develop C++ code as with other languages. I think too many people think C++ is a huge investment above other languages due to past experience, but I feel that it is just not true anymore. Thanks for this post, I think it shows just how quick it is to write performant modern C++, when you stand on the shoulders of other libraries. Just like the other languages.

Modern C++ is indeed pretty productive. It's the tooling and compile times that hold it back.

Tooling is getting better, CMake is awesome. Modules would reduce compile times significantly but it seems C++17 won't include them, so we'd have to wait a bit longer.

Perhaps you mean link time? If you're spending a lot of time compiling you are probably suffering from other problems. Most of C++'s issues with compilation come from needing to re-compile entire .cpp files because of a change in a depending .h file. If you properly forward your class declarations instead of including a header in a .h file, don't do template meta meta magic, and think about physical design a bit, your compilation time should be under control.

What I mean by physical design is, if a particular function is prone to be edited constantly, keep that function definition in it's own .cpp file.

of course I'll agree that in this day and age, we shouldn't worry about those sorts of things.

But when you do need generics and use templates, you are forced to write lots of code in .h files. There is very little you can do in that case to speed up build times, without resorting to devious things like PIMPL.

Unfortunately, even PIMPL won't help with templated code, unless you resort to manual template instantiation which is tedious.

You can usually get a noticeable speedup using unity builds, even when using something like Incredibuild. That suggests that more source files is not the answer. (I am not going to speculate as to why things are this way; it simply appears to be the case.) I'm not sure it makes a great deal of sense to separate compilation time from link time anyway (and you should also consider device reboot or program startup time as well). It all adds up to the same thing: shameful turnaround times.

This has been a problem on every C++ project I've worked on over the past 15 years, and it only seems to be getting worse. And scaled up by increasing team size, that makes it even more of a serious problem.

I hope your parting shot was intended to be that we shouldn't have to worry about this today, i.e., that we should be able to expect more from our tools, and that the current tools are shit. Because we absolutely should worry about it! This is people's time we're talking about. That isn't any less valuable than it used to be.

Often with large C++ codebases the link step can be the bottleneck. Not only it can't be easily parallelized for a single target, but even if you have multiple independent binary targets, the high memory usage of the link step makes it hard to run multiple lds at the same time.

I have seen optimized builds run faster than debug builds simply because many symbols where being removed during optimization speeding up the link step.

A lot of C++ compile time is simply generating and optimizing all the code. Modules don't help with this.

Modern C++ style generally requires making a whole bunch of little functions and classes and counting on them to be inlined and SROA'd away. This works remarkably well for runtime, but it exacts a big compilation performance hit simply because you're optimizing so much code.

Interesting anecdote, I was writing a ray tracer recently and instead of using real recursion I tried using a recursive template instead (because who needs i-cache). Basically, every bounce of a ray was a different function with <current_bounce + 1> until <max_bounces>.

The actual ray-tracing code wasn't that long at all, but if you turned max_max_bounces up (say, 100+) "code-gen" time in visual studio blew up to over one minute. That was the moment when I appreciated what everyone was saying about C++ compile times being quite slow if you (mis)use templates.

Do you have any pointers to learn modern C++?

Some resources:

The Definitive C++ Book Guide and List : http://stackoverflow.com/questions/388242/the-definitive-c-b...

I like the breakdown of books by their goals in this FAQ : https://isocpp.org/wiki/faq/how-to-learn-cpp

I would say 'C++ Primer 5th Edition' and 'The C++ Programming Language 4th Edition' are good.

And of course, Effective Modern C++, although the other Effective series books from Meyers are a must read.

http://www.informit.com/store/c-plus-plus-primer-97803217141...

http://shop.oreilly.com/product/0636920033707.do

http://www.informit.com/store/c-plus-plus-programming-langua...

http://www.informit.com/store/effective-c-plus-plus-55-speci...

Hi there, I'm interested in learning C++ but is it worth waiting for the C++17 spec to be finalised first? Will there be any significant changes in C++17?

Don't wait. Learning C++11 will put you ahead of the majority of C++ programmers. Learning C++14 put you on the bleeding edge.

After sitting down and getting everything actually working between 2000-2010, C++ now seems to be on a steady path of improvement with major releases every three years with no plans to stop.

So, it's not a matter of waiting for things to finalize. It just deciding to go ahead and climb on board the moving train.

If one is choosing C++ then performance is probably a part of the requirements equation.

These books discuss performant (and sometimes complex) C++ code - in the domain of computer graphics.

http://www.amazon.com/Physically-Based-Rendering-Second-Edit...

http://www.amazon.com/Real-Time-Collision-Detection-Interact...

About how to use the new threading standard: https://www.manning.com/books/c-plus-plus-concurrency-in-act...

Stroustrup's C++ book is good - the tour in the early part of the book covers many aspects quickly but the rest of the book is helpful reference for explaining parts that are only briefly touched on in the first bit (or if you don't quite "get it").

Also, the rest of the book is well written and worth reading for completeness, if not rather lengthy.

I agree. I got websocket server and client apps running in modern c++ in just a few days of tinkering in my free time. It took no longer than when setting up servers in Clojure, which is quite a productive language as well.

I think the crux of the development time arguments is that alternatives (eg Django, Rails) are often monolith projects that let you do everything (database modelling, controllers and templating) all in a single convenient, well documented and thoroughly battle-tested library.

You highlight the daunting prospect of moving to C++. You need to pick half a dozen separate libraries and glue them together. Sure, once you've done that a few times it's pretty quick, but it's a scary prospect for a Django dev who can program in C.

It's refreshing to hear other people think the same thing. Most of development time isn't spent physically going code. By picking C you get a high performance layer which can save a lot in infrastructure headaches and has the best performance analysis tools like vtune.

That said I've found nodejs slightly nicer for distr

Just curious, how is nodejs nicer for distribution than a binary file?

Ah bummer, looks like my response was cut off :(

If you can bundle all your dependencies into a static file you're right, C/C++ is actually better as you have no install requirements.

However in terms of maintaining your dependencies, nodejs can be pretty great with package management coming with the npm tool. When you want to share your code with the world, and for someone else to actually use it, nodejs is easier.

In C/C++ you can use CMake to have external dependencies on git projects which is the best I've seen that you can do. If you know of any other tools that help with these problems please tell!

For network-facing software, though, the memory safety problem is a huge one. It's one you just don't have with Python, PHP, Ruby, JavaScript, and Java. And I don't see any solution to it in the near future.

Does modern C++ have a decent story for writing nonblocking concurrent code yet? Something that could compete with async-await in C# or monads in functional programming?

Well, threads just became a part of the standard with C++11. As to how to use them, there is a decent story, you just have to type it yourself :)

https://www.manning.com/books/c-plus-plus-concurrency-in-act...

await is coming, maybe even in C++ 17. Visual Studio already implements it. Way more efficient than the C# implementation too (coroutines)

Promises are already there.

Promises are not really there. The C++11 version of std::future is missing future::then, which makes it useless for most concurrent tasks. You can launch subtasks on other threads and wait for the result with this future implementation, but you can't asynchronously wait for results of tasks in the same thread.

This will also only be fixed in C++17. Boost future already implements more of this, but afaik it's still flagged as experimental, so it could change again. And afaik there were also some pending discussions about how futures should interact with schedulers/executors.

Not really familiar with async-await but perhaps std::future and std::promise are helpful?? Less low-level than threads to some extent.

A networking library based on Asio has been proposed for C++17. In the meantime, use Asio.

http://think-async.com/

http://think-async.com/Asio/AsioStandalone

!! I've always heard that Asio is great, but never bothered to look because I don't like introducing Boost's 450 megs of compiler-test-suite-worthy C++ to my >45kb programs. But, as a stand-alone library. That's interesting!

I'd recommend to take a look at Seastar: http://www.seastar-project.org/

Briefly: "Seastar is event-driven and supports writing non-blocking, asynchronous server code in a straightforward manner that facilitates debugging and reasoning about performance." -- http://www.scylladb.com/2015/02/20/seastar/

Concurrency model: "Seastar futures/promises/continuations (f-p-c) are a subset of reactive programming.

Seastar performance derives from the sharded, cooperative, non-blocking, micro-task scheduled design, and f-p-c are a friendlier way of feeding tasks to the scheduler.

Seastar’s f-p-c do have some optimizations relative to other f-p-c designs. They trade off thread safety, which is unneeded due to the sharded design, for scheduling efficiency, and have a very low memory footprint." -- http://www.seastar-project.org/faq/

If you'd like to find out more, here are a few links with more information (including examples and tutorials):

- https://github.com/scylladb/seastar/wiki

- https://github.com/scylladb/seastar/blob/master/doc/tutorial...

- http://www.seastar-project.org/futures-promises/

- http://blog.cloudius-systems.com/2015/04/29/seastar-tutorial...

- http://www.slideshare.net/TzachLivyatan/seastar-sayeret-lamb...

In terms of practical applications, ScyllaDB (fully compatible--and significantly faster--drop-in replacement of Apache Cassandra) relies on Seastar: http://www.scylladb.com/

ScyllaDB has been discussed here before: https://news.ycombinator.com/item?id=10262719

I would love you to show us the C++ equivalent of this trivial PHP code:

    <?php
    $data = json_decode(file_get_contents('data.json'));
    foreach ($data as $obj) {
        echo $obj->x . ',' . $obj->y . "\n";
    }

That's the entire script.

Take a look at JSON for Modern C++: https://github.com/nlohmann/json

Perhaps this example (benchmark, actually) is close enough: https://github.com/nlohmann/json/blob/master/benchmarks/benc...

Note that the above also performs mini-benchmark with multiple iterations, dumping parsed data to another file, and finishes with a clean-up of the aforementioned dump target file.

The core (read, parse, write) can be simplified to:

  std::ifstream input_file("data.json");
  nlohmann::json data;
  data << input_file;
  std::cout << data;

No no no, this won't compile. Write the complete code, with all the includes, main function, etc. And write it correctly, iterating over the data, accessing the object fields.

There is _absolutley_ no value for a language to provide handy built-in functions that are trivially made available by a library.

If we were talking about language constructs/semantics or flow-control abilities, I'd have a very different opinion - but not having a json decoder builtin, is no reason to use or not use a language.

Don't dodge the question. Use any available library you want, just type the code that will compile.

All right, then: This one is complete (compiles, runs, iterates through the data, accessing only fields "x" and "y"):

  #include <fstream>
  #include <json.hpp>
  
  int main()
  {
  	std::ifstream input_file("data.json");
  	nlohmann::json data;
  	data << input_file;
  	for (auto obj : data)
  		std::cout << obj["x"] << ',' << obj["y"] << '\n';
  }

I have settled on Lua and C/C++ as my principle language/environment and it is a real joy. I think anyone who is 'into' C++ and hasn't yet grok'ed what Lua can do for them is going to have a good time if they dig deeper. Lua can do anything those other languages can do. And you can write C++ for the hard stuff.

I'm sort of surprised it doesn't come up more in this environment, but modern C++, Lua, and LuaJIT .. these things appear, to me, far more valuable than given by the hoipolloi ..

The point is, an investment in C++ does not mean you can't 'have all the fun toys too', because .. you can. And boy do they kick ass.

I don't understand the point of saying something like:

> "I show how to build a modern C++ high-performance, asynchronous echo server that can be written with just 48 lines of code."

The fact that it is 48 lines of code is almost meaningless. If the framework was a different design it could be done in 1 line of C++ code or 1 line of COBOL. Given the right library/framework, any application can be done with 1 line of code (taking a bit of poetic licence here, but hopefully you see my point).

As a discussion about using Wangle in C++, the article has far more merit.

Perhaps I am just being overly pedantic.

I agree. Especially because this sort of code seems to be the whole purpose of this framework. I would expect it to take fewer LOC.

> The fact that it is 48 lines of code is almost meaningless. If the framework was a different design it could be done in 1 line of C++ code or 1 line of COBOL. Given the right library/framework, any application can be done with 1 line of code (taking a bit of poetic licence here, but hopefully you see my point).

I think it's fine to state that; are you ever going to write C++ code with zero included libraries? Probably not. If you write some string manipulation code and you boast that you did it in 10 lines are you going to count it against them that they included std::string?

I think saying how many lines of code it took, using the Wrangle framework, shows that it can be powerful and easy to use versus many of the libraries you end up seeing in C++.

If those frameworks/libraries are battle tested and not one off solutions for the project then I do think it has merit.

Imagine building an overpass out of 3 prefabricated concrete pieces; two columns that serve as the base pedestals and 1 beam that spans them. The 48 lines of glue code to me are the nuts and bolts used to tie the pieces together. If the nuts and bolts are structurally sound then I would say our overpass has merit..?

I'm far more interested in what those 48 lines of source actually compiles to, or how much framework code is involved. I've found that the more of the latter there is, the more difficult debugging tends to be.

I agree that a more fitting line would have been, "I show how to build a modern C++ high-performance, asynchronous echo server using Facebook's Wangle (which comes out to be only 48 lines of code).

However the TITLE OF THE ARTICLE is "Writing modern C++ servers using Wangle." Furthermore, the author states that he's using that framework almost immediately after the statement you quoted, so I really see nothing to complain about. It would be more of an issue if the author led with a huge introductory paragraph or click-baity title that failed to mention the library she or he is using.

Knowing how many lines it took for the author to implement the server in the Wangle framework is very relevant to the reader.

> 1 line of COBOL

Offtopic, but this would be quite a challenge.

it may not be an information-dense sentence, but it is merely a single sentence. one which you've isolated from it's context.

It is the direct TL;DR quote from the Author himself, given prominence with blank line separators, italics, bold prefix and located near the very start of the article, so hardly out of context.

48 lines is totally relevant. If I'm coming into the article and thinking, "Oh god, C++? This is going to be a mess," then seeing that it's only 48 lines is a great hook.

> The fact that it is 48 lines of code is almost meaningless. If the framework was a different design it could be done in 1 line of C++ code or 1 line of COBOL.

Yes, you can optimize it to whatever, but the point of mentioning that it is only a small number lines, it informs the reader they aren't going to get lost in a tone of code.

For what it's worth, the network I/O model design+implementation is rarely what makes the difference between high performance, or not, 'servers' -- and that is a mostly solved problem, at least in terms of patterns and practices that are adopted by 'modern' servers (few threads, no more than actual hardware CPU cores, async.multiplexing, etc).

High performance is about everything else, including processing actual incoming requests efficiently, reducing block/busy time in threads, caring a lot about memory access/caches. They key IMO is concurrency and migrating blocking operations to other dedicated threads, and this is where an efficient coroutines implementation would help both with accomplishing that task, but, perhaps more importantly, keeping the programming model simple.

C++17 will introduce support for reusmable functions ( http://blogs.msdn.com/b/vcblog/archive/2014/11/12/resumable-... ). This will be a game changer. Now, you can approximate this by e.g creating task abstractions and use lower-level(but slow) setjmp/longjmp functions, or just design tasks/functors that hold state that can schedule other tasks in turn, and so on. It works, but the overhead, mental, but also n terms of processing/executing/scheduling can be too great.

>I must warn that it unfortunately doesn’t build on Max OS X as of yet so I recommend to virtualize Ubuntu 14.04 to install it.

Important info for some folks (i.e. myself).

I stopped reading after reading the first line of code containing a virtual function and smart pointers. Nothing is performance oriented in this post. It's just a plain old dummy TCP server like anyone would write.

I half agree; it's easy to be cynical. Virtual functions can be a terrible idea... if they are in a hot path.

Is EchoHandler::read such a path? Maybe, you really can't say without profiling. If anything using std::string is the red flag for me.

no benchmarks. what gives ?

Yeah, and no mention about boost::async, which is in Boost, ie almost standard.

You mean std::async (c++11)? http://en.cppreference.com/w/cpp/thread/async

oh, I mean ASIO.

Folly, the library underneath wangle, is running on top of boost::asio, so this is basically an abstraction on asio two layers up.

ASIO header-only implementation is also available without Boost, fyi, as a standalone library; the same one that's in boost, but used without boost.

Folly does not use boost::asio, but rather libevent:

  libevent is an excellent cross-platform eventing library.
  Folly's async provides C++ object wrappers for fd callbacks
  and event_base, as well as providing implementations for
  many common types of fd uses.

https://github.com/facebook/folly/blob/master/folly/io/async...

grepping the codebase for `asio` brings up nothing.

Interesting. Thanks for the correction. I had seen the boost dependency and assumed it was using asio under the hood I guess.

You mean LOC vs another framework? That would be the only relevant metric in this context.

It says "high-performance servers" in headline...

I love to read stuff like this, "with a bit of effort I was able to build the whole back-end for my startup using C++", then I start to think: what would it take for me to migrate off Golang to C++?

I love to develop in Mac and deploy in Linux/Docker, so whatever stack I pick has to be compatible with both OS's, with this in mind I made a list of libraries I'd need to move my API to C++:

* Web Server library

* Web Framework-ish library

* JSON library for REST interface

* Database client library for my database

* Integration testing

After reading these useful blog posts I guess I'll pick:

* Web Server => Facebook's Proxygen

* Web Framework-ish library => Facebook's Wangle, but it doesn't seem to work in Mac, that's already a problem, I don't like having to have two dev environments.

* JSON library => looks like Facebook's Folly seems to have something for it, let's hope it uses modern c++ constructs and not some obscure templating magic to make it faster than other implementations.

* Database client library for RethinkDB: RethinkDB has no official driver and the only community driver public repository has only 22 starts and no CI setup on Github, I'm reluctant to trust this code, this is already a deal breaker to me.

* Integration testing => Wangle's documentation is scarce, there wiki is empty and the repo doesn't have a single code sample to take a look at. I can't go to production without testing, another deal breaker.

Yes, I agree, I don't have to pick RethinkDB. MySQL and PostgreSQL c++ drivers are very mature to use in production apps.

Of course they are, but:

* Do these drivers use modern C++ constructs?

* Would these drivers block proxygen/wangle IO loop(in case they have one?)

* Would I have to use callbacks to make achieve full IO speed? I hate callbacks, there is a reason I abandoned Node.js a long ago.

* How do I put all these dependencies together?

As with Golang, as much as I hate it's lack of generics and proper inheritance:

* Web server: Standard library `net/http`

* Web framework-ish: I don't really need a framework/pipeline for it, there are some neat web routers out there that just help me do the job.

* JSON library: Standard library `encoding/json`

* Database client library: gorethink and almost all the Golang database clients out there have significant adoption with over 700 stars on Github.

* Integration testing: Standard library `net/http/httptest`

Unlike D, C++ and Rust; the Golang ecosystem is unified, I know I'm able to pick a client library and it'd work with whatever stack I have in place because there's only one scheduler and therefore only one way to do IO, plus it's non-blocking without callbacks. I'm not a fan of vendoring dependencies, but it simply works.

In conclusion, I love C++ and even with all the significant progress made in C++11 and C++14, I'm still stucked with Go because it just works.

I also run into this problem all the time. I hope that with the networking TS standardizing on basically boost Asio and the incorporation of coroutines into C++ we should start seeing some unification across all network/do client libraries in terms on non-blocking use.

That's a lot of code.

Did you test your server with 1.000.000 connections?

How about input fuzzing? Did you try crashing it and see how quickly it can recover? Unit tests? What about zero downtime upgrades? DDoS resiliency? Can it utilize all cores on a machine? Scalable? What are the median and max response times under load?

Oh it's just a quick demo? Nevermind.

Title needs to be edited to: How to quickly write a basic, modern C++ using Wangle.

The API looks a lot like Netty

I enjoy seeing Facebook libraries and Google libraries in the same program.

It's nice that google are using snake case for library filenames and facebook are using camelcase. :)

Example:

#include <proxygen/httpserver/HTTPServer.h>

#include <grpc++/server_context.h>

Reminds me of Doug Schmidt's ACE Reactor...

Any classical logic is possible, because C++ is a Turing-complete language.

That said, of course, how much you can actually do and how fast you can do it depends a lot on the hardware you have available. Not every computer can run fast, high-quality graphics, no matter what language you coded in.

THAT said, C++ is considered a relatively high-performance language and is often used for things like advanced game graphics.

Reply on news.ycombinator.com