Wangle – an asynchronous C++ networking and RPC library | Hacker News Reader

Wangle – an asynchronous C++ networking and RPC library

code.facebook.com

164 pointsdoppp74 comments

I have used this library pretty heavily in some of my recent work and I love how easy it makes building performant async network servers in C++. Minor nitpick: It would be nice though, if they built shared libraries by default though, last release I took I had to tell cmake to make a shared library. Since this is used by fbthrift as a dependency, I think it really makes sense to build it that way by default. I also use fblualib and it builds with a very old version of some of these libraries probably because the FAIR team didn't want to mess with tweaking the builds so they can work with standalone wangle (parts of the library used to be part of folly).

After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down.

> often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down

I don't see how that's an argument against a language being "superior to C or C++". You can't create a self-hosting compiler for your language without having already created a non-self-hosting compiler for your language. That means every language is going to have to be "bootstrapped" in terms of something else first. C is the usual candidate just because there are a plethora of useful compiler/runtime/vm-ish libraries people like to rely on for "language prototyping" that expose C bindings—and there are very few other languages than C/C++ that have zero-overhead, zero-friction C binding support.

Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead.

> Rust is one of those languages—and it does indeed look like things are going in the direction of new languages getting prototyped/bootstrapped with a Rust compiler/runtime/vm instead.

I'm curious what examples you have in mind here? LLVM seems to be the default go-to compiler infrastructure, at the moment.

> After most compilers started properly supporting the newer C++ standards, it seems there has been an increase in the amount of love that C++ libraries are getting. Sadly there are many more compilers out there than just gcc, clang, icc and msvc++.

Many of those aren't still C++11 compliant, specially in mainframes, classical commercial UNIXes, embedded real time OS and certified compilers.

But it is true, thanks to the productivity of latest standards, there has been an increase in C++ visibility, specially thanks to Microsoft.

People like to bash their C++'s compliance, but they are the commercial vendor with the more up to date support, even than icc last time I checked.

Also they were the main sponsors for Back to native conferences and eventually making it together with CppCon.

https://blogs.msdn.microsoft.com/vcblog/2014/04/03/cppcon-th...

Also Windows Phone is the only mobile OS where C++ is a first class language.

>. A lot of people like the to say that [x] is superior to C or C++ - often ignoring that their favorite language's libraries/runtime/compilers/vm is most likely implemented in the very same languages they are putting down.

A lot of people like to point out that fact, ignoring that many compiler writers tend to use C or C++ out of convenience for existing tools instead of bootraping the full eco-system around a new programming language.

Many use the same route as Go, using one of those languages and when the toolchain gets mature, those parts get re-writen.

All in all, I do like the language, now if we could get rid of those C underpinnings...

Intel's C++11 support continues to be quite buggy. I'm still running into with issues with Intel's support for auto type deduction and for uniform initialization in non-obscure use scenarios.

Sorry I was updating the comment as you replied.

I never used it, but last time this discussion came up and I searched for their support page many C++11 features were either TBD or GNU/Linux only.

Thanks for the feeback.

Windows phone, first class C++?

Man the times have changed. I remember when they made native code verboten in the Windows Mobile -> Windows Phone transition. Killed off a whole product line I was working on at the time.

Yes, you can either use C++ plus the C++/CX extension, which although look like C++/CLI are actually native.

Making Visual C++ GUI developer experience finally comparable to what C++ Builder was offering in the mid-90's.

Or if C++/CX extensions is something one hates, there is the Windows Runtime Library, a template based library similar to ATL.

Finally Kenny Kerr has created a C++ projection based in pure modern C++ ans has been eventually hired by Microsoft to carry on his work.

https://kennykerr.ca/2015/10/02/microsoft/

iOS comes second in terms of C++ support in mobile OS, where Objective-C++ plays the same role as C++/CX on WP.

Just Android forces developers to enjoy the wonders of JNI, with APIs like Skia only exposed through it.

I know that there are a few other mobile OSes that do support C++, but their market share coupled together is even worse than WP.

"Also Windows Phone is the only mobile OS where C++ is a first class language."

Umm... I don't know how you define "first class language" but our SDK (networking stack) compiles C++ code right in the same XCode project with ObjC code without any sort of translation layers etc (a la JNI in Java) between the two.

iOS doesn't expose the majority of its APIs as a C++ APIs, they are either plain C, or Objective-C.

The SDK you mention is a code example, that is the only support iOS offers for C++, meaning infrastructure code with little interaction to the whole iOS stack.

If I am not mistaken (no iOS dev), besides the IO Kit and Metal shaders, there isn't much more in terms of C++ APIs in iOS.

You have to wrap calls to Objective-C via Objective-C++, for a full stack app. With Swift slowly taking the place of Objective-C, this will be even harder.

Whereas WP offers the so called projections as C++/CX (similar to using Objective-C++ in concept), C++ with the WRL template library, or plain modern C++.

All of those projections offer 100% of WP APIs as C++ APIs.

Although actually the majority of WP developers just makes use of C# alongside C++.

Although I agree with most of the comments you said, I have to strongly disagree with "First class C++ support of Windows Mobile" when compared to iOS (or even Android). I have an entire non trial stack of C/C++ libraries that I use which I easily bring to Android and iOS using a cross-compiling build system we wrote (https://github.com/amigocloud/amigomake). The C/C++ dependencies include :

Freetype v. 2.4.10, OpenSSL v. 1.0.1m, Bzip v. 1.0.6, libidn v. 1.24, GEOS v. 3.4.2, CURL v. 7.39.0, Proj4 v. 4.8.0, SQLite v. 3080700, Jpeg v. 8d, PNG v. 1.2.50, Minizip v. 11. uuid, libwebsockets, rapidjson, Spatialite, Coffeecatch, tinydir, Freetype-GL, libtess2, RTree, Spline, SHA, lodePNG.

and are used by this cross-platform mobile app: https://github.com/amigocloud/amigoclient

Getting this libraries to work on *nix platforms (e.g. Android or iOS) is easy. Getting them to work on Windows Mobile is a nightmare - starting with the fact that they don't even support things like OpenGL...

Those C and C++ dependencies are using OS specific APIs.

The developers not caring about writing portable code doesn't make it less first class on Windows Phone.

I don't see any chapter on ANSI C or ANSI C++ standards about OpenGL.

When I learned C and C++, targeting UNIX only didn't meant much, because POSIX wasn't the same thing across all UNIX systems and Windows wasn't the only non-POSIX OS.

So we learned to abstract the OS APIs properly.

Nowadays it seems we are reaching a UNIX mono-culture.

I have been developing code for 20 years using C/C++ - it sounds like you have been doing this even longer.

I think it is completely unfair that you simply dismiss and blame the libraries "because the developers did not care and wrote it without portability in mind".

I am not sure what your own personal definition of "first class citizen" is, but as such, IMHO it doesnt even make sense to argue it.

All I can tell you is that I can take all these projects and create a Windows 7 build that works - so the work to make it "portable" is there (even with all the nonsense of making sure they are compiled against the exact multithreaded vc runtime version and other windows quirks). MS has specifically chosen to not support other subsystems (e.g. OpenGL) that make portability easy/realistically feasible. To me, having a "hello world" work doesnt make a platform a "first class" system - bringing complex well known libraries and make them work does. That makes a first class citizen. I know we disagree - so we'll just have to agree to disagree :)

I started coding in early 80's.

First class citzen for a programming language means:

1 - It is part of the OS SDK

2 - All the SDK tools support the language, including RAD tooling for UI, Web, database, ....

3 - The OS APIs are written with this language in mind

4 - The OS API documentation uses examples in the programming language.

So C, Swift and Objective-C are first class in iOS and Mac OS X.

JavaScript is first class in the browser

C is first class in UNIX

C#, VB, C++ are first class in Windows

Java is first class on Android.

What open source libraries are supported on a OS don't have anything to do with first class support.

> What open source libraries are supported on a OS don't have anything to do with first class support.

Under your definition that is not true. Under mine, the ecosystem of libraries you can use on that platform is a key factor.

I just tried to compile folly in OSX and it fails to compile with homebrew. https://github.com/facebook/folly/issues/400

Also, Once I manage to compile folly and then wangle, what kind of database drivers can I use in conjunction with this? Would I shoot myself in the foot if I use this with libpq under CPUThreadPoolExecutor?

Same with Rust and dlang, I'm not sure how to apply the freedom to have pluggable executor into something useful in real life, I think this is the main reason Golang is killing it and it keeps getting better.

For now I have my money on Go but can't wait for http://www.ponylang.org/ to become stable.

Edit: Issue reported has been closed, I can now proceed to install wangle https://github.com/facebook/folly/issues/400

Edit 2: Found another issue, this time installing wangle https://github.com/facebook/wangle/issues/37

Go uses synchronous I/O backed with an M:N implementation. Rust and D use synchronous I/O backed with a 1:1 implementation. Semantically, there is no difference between the two. The difference is in the implementation, and 1:1 threading is not as slow as you think on Linux.

Why are we talking about Go, Rust, and D on a thread about a new C++ library? I mean, I know the answer, but I wish we weren't.

I ordinarily wouldn't care, but I usually feel like I have something to learn from ambitious networking libraries even when they're written in languages I prefer not to use.

For what it's worth, I try to not bring up Rust in threads about other languages; it just happens that most threads about Rust appear in threads about other languages.

I would rather talk about the implementation of async I/O too.

I just wanted to point out the barriers a developer faces when trying to use C++. Sorry, I can't resist compare it to what I know, which happens to be other programming languages different to C++.

Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)? Conventional C++ wisdom advices against it. But then it may be informed by some old and bad pthreads implementations.

A reasonable compromise is to use a threadpool (which is a big part of what comes with wangle) with a fixed number of threads and you queue the tasks to be run by the threads. It works quite well in practice, and you can use coroutines (I'm most familiar with the implementation in boost coroutine and the Ewan webserver personally) to go the whole hog and get something as lightweight as Erlang green threads or goroutines. I would only bother doing that of course if in your particular use case you find the threadpool doesn't perform well. With proper sizing and broken up units of work, it really works quite well in practice.

Usually memory usage ends up being the limiting factor, not thread spawning performance. Memory usage is independent of 1:1 vs. M:N, though, as pthread stack sizes are configurable.

> Does that mean that one can use system threads as frivolously as goroutines (e.g. spawning new thread for each incoming request)?

Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call.

> Conventional C++ wisdom advices against it.

With 32 bit CPU's you might run out of virtual memory quickly, since each thread allocates virtual memory for its stack.

With 64 bit CPU's there are no issues. (Also never forget that virtual memory isn't real memory, it's just a page counter.)

> Yes. In fact, under Linux oftentimes spawning a thread and then running a system call can be faster than just running a system call.

Interesting, do you have any further info (references, cites) on this?

Sign me up for wanting more info as well. I'm curious how that could work out, since spawning a thread is a system call.

> what kind of database drivers can I use in conjunction with this?

Usually synchronous database drivers have to be reimplemented in an asynchronous fashion to work with event loops. It's not very hard though, could be as simple as adding jump tables and context structures into synchronous functions.

Emulating asynchronous execution with a thread is ok too. You do that to access filesystem anyway.

But if your first thought is about accessing a database server - you probably don't need to go asynchronous at all.

> Emulating asynchronous execution with a thread is ok too

Straying away from the topic, but I used to support a file streaming library for the game studio I worked with, and while this approach worked, some things were not gracefully handled - for example canceling reading in the middle, or somehow giving all your I/O requests to the underlying system, so it can decide what's the best way to read them (aio?).

The wangle native way of doing this is just to use an IOThreadPoolExecutor: https://github.com/facebook/wangle/blob/master/wangle/concur... Just make a block of threads to run your db tasks, and you can use a promise/future to return the results back to the async code.

There's https://github.com/facebook/squangle for MySQL, though it's lacking documentation, and requires a https://github.com/facebook/mysql-5.6 or WebScaleSQL client library

MariaDB has an asynchronous C API.

libpq (Postgres's client library) has an asynchronous interface as well.

The problem is the exact opposite. The question you should be asking, is: how can I use my go code in Rust/D/C++? The answer is, not very easy, given that go's runtime is largely incompatible with C.

At least with C++, you can easily write C shim layers and your code becomes compatible with most runtimes in existence. The opposite is not true with Go.

The first thing that comes to mind after reading this is GRPC (https://github.com/grpc/grpc). That library has the advantage of supporting just about every language, not just C++.

Good to see C++ is gaining more visibilities these days, considering the Oracle/Java combination I hope C++ can take away some JAVA dominance, at least nobody is going to sue you over C++ APIs.

Failed to compile with CMake 2.8

    CMake Error at /usr/share/cmake-2.8/Modules FindPackageHandleStandardArgs.cmake:91 (MESSAGE):
    REQUIRED_ARGS (missing: FOLLY_INCLUDE_DIR  OLLY_LIBRARIES)
    Call Stack (most recent call first):
   /usr/share/cmake-2.8/Module/FindPackageHandleStandardArgs.cmake:252(_FPHSA_FAILURE_MESSAGE)
   cmake/FindFolly.cmake:23 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
   CMakeLists.txt:17 (find_package)

Had to look up the meaning of wangle. "obtain (something that is desired) by persuading others to comply or by manipulating events."

Does anyone know where facebook is coming up with all these words - folly/wangle?

Folly is a common word, I think. As non-native English speaker I know the meaning of folly. But I saw wangle for the first time.

Yup, I know folly as well (non-native speaker as well). But it doesn't make sense to name a library 'folly' does it?

What I was wondering is if this some self-deprecating humor or these words are popular phrases from some popular book/meme/play.

I was going to correct you and say you meant "wrangle". Turns out they can have the same meaning! Wonder if there are other examples of two words being one letter off that mean the same thing.

"Folly" is a relatively common English word. "Wangle" is less common but I still see it from time to time.

What I find more interesting than the naming is FB's willingness to remove features from Folly as they are made redundant by Boost or std. I'm unaccustomed to such a lack of arrogance from a major player.

Doesn't removing features break backward compatibility?

My favorite C++ library is Qt; they never remove public API for that reason. In Qt world, we update regularly to get the bug fixes and other features. Last thing I want to do is port my code to some other new API to update the library.

If it's a feature that has made it into the std/Boost libraries, we'll be talking atomic utility functions which are being removed, in favour of calling the std/Boost ones. Not actual API-level features.

beats me i wonder the same thing all the time

"folly" is a play on Facebook Open-source LibrarY. I think using uncommon English words helps with searchability, although of course is not as good as made-up words or acronyms.

When I name things I look for a word that doesn't get drowned out by unrelated hits in search engines, and has some kind of link to what the library does. Straightforward names are great when they are obvious, easy to remember, and not a search disaster. Otherwise clever memorable names make a pretty good substitute.

I picked the name Wangle because it is a synonym to Finagle, and because I couldn't think of a simple self-descriptive name. I think I also had the Edward Lear poem in the back of my subconscious.

We don't have a central naming authority - people just name their projects whatever they want. Sometimes we have to or want to rename things for various reasons but it's mostly similar to the open source world.

https://www.urbandictionary.com/define.php?term=Wangle

There are many meanings, including:

> A combination of the words wang + angle. > Noun: A curved/bent/crooked penis.

So Wangle is (mostly) a protocol agnostic platform for building asynchronous network clients and services in C++.

What transport layers are supported and can it be extended?

What I can gather, it only supports IPv4/IPv6 TCP or UDP transports. TLS is also supported.

Notably, local pipes seem to be unsupported. Those can be sometimes useful for performance and security/isolation reasons.

Also, check out crossbar.io and the WAMP protocol, offering transparent, clean and yet simple PUB/SUB & RPC accross languages, including C++, Python, PHP and JS (node and yes, browsers).

WAMP is a protocol, Wangle is a framework that you can use to implement protocols (like WAMP), crossbar is a broker (an application that implements the protocol using a [python] framework. So it's not a direct comparison.

does anybody know why this library is not based on (boost::)asio ?

I'll take a shot in the dark and say they probably value their compile times. Every boost library I've ever used has the tendency to pull in the world from a single header and take forever to compile.

ASIO can be used standalone with any C++11 compiler and the compile times are quite good.

If I should make a guess then I'd say that the original author of that part of the library either started before ASIO was standalone and/or simply has a hatred for boost (which would be unwarranted), or he/she was simply more experienced with libevent...

IMHO it's quite sad that they didn't use ASIO though as it's far more extensive.

That's the thing about reputation though, really easy to burn and really hard to earn back.

I've worked on projects where touching a main header incurred a 1 hour compile even with Incredibuild. You can understand how people can get sensitive with compile times in cases like that.

And interestingly, Chris Kohlhoff (asio author) is currently working on an executors library / c++ standard proposal.

Other question: how would wangle compare to Cap'n Proto (as the post mentions that wangle is an RPC framework, and is zero-copy)

Zero-copy in capnproto docs and in this blog post actually mean different things (the first means that serialized and in-memory representations of capnproto messages are the same and the second means the absence of copying between user space and kernel space when sending a file over the network).

I haven't used the standalone version, but I have used the boost version for build a cross platform RPC library quite intensively. Unfortunately compile times there were really bad, for about 10kLoC C++ I had a compile time of nearly 1 minute (about 5years ago on a medium speed machine back then). I later also built the same library using Qt and the compile times were below 20 seconds. I think the differences came from the fact that asio is heavily templated, you get all in headers and favors directly instantiated objects instead of pointers. In QT you mostly have pointers (and therefore work with forward declarations to the implementations).

Besides that I think asio is a really good library. It gives you LOTS of options how to handle your workload. E.g. much more than libuv, in the sense that you can mix asynchronous and [non-blocking] synchronous IO calls inside your application, do IO calls from different threads, organize work with executors (io_services) and strands, etc. The drawback is that not all of these patters work good, and giving the users this kitchen sink of functionality could lead to worse results than a more focused approach. E.g. it took me a while to realize that calling io_service.run() from multiple threads is not the best idea.

They actually are pulling in boost. System thread and filesystem but not asio, according to the cmake file.

The other commenters are probably right that they wanted to integrate with folly's existing async infrastructure.

Companies tend to stick to their own ecosystems, and in this case Facebook's folly library has already bought into libevent, for better or worse. Hopefully once asio gets standardized more libraries will switch over, but until then there are a number of legitimate issues:

* There's a Boost version and a stand-alone version, you might want to use one but not the other: * Boost requires linking against Boost.System, making it not header-only anymore. * Stand-alone isn't as widely used and if you've already bought into Boost, you might not want an extra dependency.

* Asio's API documentation exists, but that's about the best you can say about it.

* The standard version is likely to resemble the library but won't be quite the same. When it gets standardized, you still have to adapt your code. One might reasonably argue it doesn't matter much whether to use one dependency vs. another if neither binary nor source compatibility can be preserved.

* The Networking TS didn't make the standard for C++17 so it'll still be a long time until the C++ world converges around it.

* Chris Kohlhoff's complete absence from both the Boost and stand-alone GitHub project's issue queues raises questions about the state of maintenance, there are lots of pull requests pending without upstream action. (The one email I sent him regarding his executors proposal also never got a reply.) Modifying their own library would be much easier for Facebook than getting asio patches upstream.

Personally I'd like to see a world where new C++ projects depend on asio, but it's understandable if others feel differently about it.

"Wangle is heavily influenced by the JVM-based libraries Netty,"

Stopped reading there. Have experience with using Netty in a java server application and engineers dread being assigned a bug or enhancement in that codebase. There's a tendency throw around the word "over-engineered" lightly but netty is a real, live and inexplicably popular example of exactly what that word means.

I like that they explained the reasons behind the design decisions of the library. Looking forward to the post about the RPC part!

I'm wondering, how does C++ fare against Nodejs now, if this library is implemented for creating servers?

Node JS unfortunately doesn't have the notion of separate IO and CPU threads.

would this be useful for game development?

I can't speak for Wangle, but I've used Google's version (GRPC) to implement the networking transport for an N64 netplay hack[1] I wrote, and it was awesome.

[1]: https://github.com/water-works

Good that they mentioned Netty as inspiration

In 2016 I would write networking and RPC in go.

Me too - and I don't even know C++.

I was joking that the main reason for dismissing C++ for this task so quickly was not knowing it, and when and where C++ is used.

How did I came through, to be so unvoted? I'm confused.

"networking and rpc" must be written in the languages your server and client are written in, (or something compatible).

Writing a networking library in Go does nothing for Facebook if their services are written in C++.

Because you want to spend 10x as many CPUs?

Mostly because it's easier, performant, and because of the nature of the language it's less likely I would make a mistake doing anything in the language compared to C++ or C.

Reply on news.ycombinator.com