Folly – Faceboook’s open source C++ library | Hacker News Reader

Folly – Faceboook’s open source C++ library

github.com

110 pointscrakenzak87 comments

Interesting how any company with enough C++ developers will eventually develop a huge library like that (with data structures and tools and system wrappers). In the JS ecosystem, we may complain that node_modules is large, but at least we are not reimplementing and rewrapping the wheel at every corp (same for python).

Maybe if C++ had a standard package manager, people would stop coding huge libraries like that, focusing on smaller libraries that can be reused (independently).

I feel a bit like there is some kind of programming totalitarian personality that wants there ever to always to be one thing.

There are so many build systems in C and C++ because for each of those, someone finds something to be inadequate and has something else that works for them.

Just like there is no standard for sizeof(int) or whatever, because it started with one size on one CPU and somebody brought it to a different CPU where another size made better sense. Today you can usually guess that it's 32 bits, but there are domains where it's not.

These languages are meant to not impose a lot on the implementer or practitioner. Having the standard specify that every build needs to potentially pull down a bunch of crap from github or wherever else is going to be really annoying to a lot of people. It is a misguided suggestion. I think it mostly comes from people not working in the language, or unfamiliar with code bases that are working well as-is.

> Just like there is no standard for sizeof(int) or whatever, because it started with one size on one CPU and somebody brought it to a different CPU where another size made better sense. Today you can usually guess that it's 32 bits, but there are domains where it's not.

This is different: C has to be implemented efficiently on a wide range of hardware, almost without regard for how good of a fit that hardware is for C or any modern language. Machines with 36-bit integers and 18-bit pointers? They got C compilers. Machines with no native notion of byte addressability? Plenty of C compilers for those things. Machines with application-visible segmentation? Of course! Those machines were quite popular for a while.

Therefore, the C programming language has to at least try to make it possible for portable programs to exist. That's the implied contract of the C standard: Code to the standard, and your programs will be compilable and runnable on every platform targeted by a conformant C compiler. So, since int is going to be a size adapted to the hardware, as required for efficiency, the programs have to know how big an int is, as required for portability.

> Having the standard specify that every build needs to potentially pull down a bunch of crap from github or wherever else is going to be really annoying to a lot of people.

No language standard specifies that, I'm pretty sure. The ES6 standard doesn't specify left_pad and the C++ standard doesn't specify Boost. However, the standards also don't specify things which could replace those external libraries in a portable fashion which is guaranteed to always be there. If you take something out of the standard library, it has to go somewhere; people aren't going to stop doing that thing just because the standard no longer mentions it.

I said there should be a standard package manager, this does not mean that it has to be the only one. It would just be easier for everyone if there was a widely adopted way to package, install and update stuff, like most modern languages.

There is such a thing for C/C++ devs: Git.

This works so well you may have hardly even noticed its so widespread as a solution to the problems you describe.

>There are so many build systems in C and C++ because for each of those, someone finds something to be inadequate and has something else that works for them.

Makes me wonder why Lisp is cursed[1] but C++ isn't.

[1]: http://www.winestockwebdesign.com/Essays/Lisp_Curse.html

C++ has OS vendors backing it, while Lisp OS vendors lost the war.

Never underestimate the power to use the OS SDK to sell a programming language.

I would argue that any company with enough JavaScript developers will eventually develop JS libraries too. There's nothing that limits this to just C++. Facebook has folly but also react and a whole set of tools around it.

Folly is more like an addition to the standard library. For that sort of library, you want parts to be deprecated over time as the language's standard library grows. React has no place in a standard library; it's more of a framework.

That you simply ignore the fact that C++ projects generally do prefer to not build from any "framework".

Framework in JS, is just similar to a more abstracted library in C++.

In the whole stack (or spectrum) of machine, language, application, C++'s design philosophy naturally encourage things like Folly and Absl. Just like JS encourages React, VUE etc.

This is more like if every company reimplemented core-js... worth a little more examination than "large companies write libraries"

React attempts the re-implement the browser DOM API and its associated UI conventions. That's way more WTF than folly, which is basically jQuery for C++.

React attempts to wrap the browser DOM API and it's conventions in a way that's (for people who use it) easier to reason about, that's kind of like saying Anvil is a WTF because it wraps Vulcan

Also yeah, every reimplementing jquery would be extremely strange and mostly defeat the point of jquery... which is to not have to implement jquery's constructs in Vanilla JS

I don't think I agree, most don't reinvent the common parts: jQuery, lodash, angular or react

Most don't, but Facebook does have its own equivalent of jQuery, lodash and angular. A few other very large tech companies are similar.

Most C++ shops don't have their own equivalent of Folly, they use Boost or Google's Abseil or something else.

I think that's the key point, the scale in which this starts happening is the sizes of the Googles and Facebooks of the world (in C++: Abseil and Folly) but also in Java (at Google: Guava), JavaScript (at Google: closure, Angular/Polymer/Lit/and many others).

So people extrapolating from a ~100 person company or a ~1000 person company using JS might as well think it's about the language, but it's really about the size, I think.

Closure isn't so much a standard library as a typescript alternative though, I don't think the comparison works.

Look at Google and the Closure standard library for an example of a company writing a lot of JS that ended up writing it's own "standard library"-like thing.

I don't think this is a particularly interesting take. Google's utility c++ library isn't that different from eg underscore.js (or lodash)

Google uses some boost code and plenty of other external libraries (I can't speak to Facebook but imagine it's similar) - I think the point is just that Facebook and Google have an enormous amount of c++ code and have thus created libraries (some of which are fundamental enough or self-contained enough to open source)

This is just what happens when a bunch of engineers are all working in the same language - the utility libraries become large enough, robust enough and useful enough to open source

-----

As an aside, I think the similarities between folly and absl say something more meaningful about the state of c++

If you're using C++, you want speed and/or memory efficiency. Generic libraries won't give you that. You need custom implementations of data structures that make the trade-offs relevant to your needs.

There is no reason for memory or speed to be different. What I mean is that parts of these libraries could be single small packages that would be as fast.

As an example, look at this folder which:

https://github.com/facebook/folly/tree/master/folly/compress...

In short, these are compression tools. Why not have 1 easily uninstallable package per compression tool in a standard package manager that works the same on all platforms.

Things don't have to be more or less generic.

So the comment is really about the size of these libraries?

This is more of an artifact of companies that produced them, which use large monorepos, than the ecosystem itself.

That said, you can pick and choose which part of boost you use, just as you can pick and choose which part of Abseil/Folly you use.

> Why not have 1 easily uninstallable package per compression tool in a standard package manager that works the same on all platforms.

Because C++ sucks and its dependency story is a nightmare. There are a few competing C++ package managers. They’re all relatively new and none of them are particularly popular.

The most widespread C++ libraries are ones that are single header files. Because that’s the only thing that’s trivial to implement into an existing project.

Compiling OpenSSL requires installing Strawberry Perl. This type of garbage of BS is the norm.

Source: C++ dev for 15 years

> The most widespread C++ libraries are ones that are single header files. Because that’s the only thing that’s trivial to implement into an existing project.

I think this has much more to do with templates than anything else. A template can only exist in a header.

Then there is also the performance benefit of inlining. (As well as performance hit of inlining, namely code size.)

There is a separate, much later movement/phenomenon of C (not C++) developed as header-only libraries, which by contrast I don't think there is nearly as good reason for except for that some people are confused at how a linker works. I have seen use of that technique advocated on Hacker News much more frequently than in production.

Header only and single file are slightly different. A headery library may include many many files. A single file or single header may be “header only” or it may have an impl define. People who prefer this distribution mechanism are generally ok with 1 header plus 1 cpp.

These libraries generally have no third party dependencies. Because tracking down and compiling C++ dependencies is a dumpster fire.

Or maybe they don't actually need the dependencies. There is a style of writing code to minimize dependencies and many people value this.

I would take the sqlite amalgamation as an example of good use of "few files, few dependencies". First off, it is above all a solid piece of work. It is distributed with one header file and one .c, even though it is developed as multiple files. Iirc the sqlite website cites a ~10% speed improvement or so due to the optimizer being able to do a better job across the whole library. In my opinion this is a better reason than "tracking dependencies sucks" or "omg too many files".

I'm pretty sure it's the (lack of) package managers.

The presence of a defacto package manager completely changes the library dynamics.

I can't even link and test properly against GMP on Windows :/

Sigh. Keeping sqlite as an example. On most Unix-like OSs (and even windows now) there is a dynamically linked sqlite managed by a package manager.

It also predates the time period when people would whine about a lack of package managers.

As do a bunch of these libraries.

> If you're using C++, you want speed and/or memory efficiency. Generic libraries won't give you that.

The standard libraries should, in that they can have Spooky Deep Knowledge of the compiler and other standard libraries which external code won't, like how there's no way to add two ints which is faster than the standard + operator.

I think you're right that some things, such as integer addition, are quick and don't require a lot of thought, but a lot of teams leaning to C++ stems from larger-scale performance considerations that aren't one-size-fits-all.

A simple example is `std::vector`. The structure is a nice resizable array, but for a long time you couldn't override the allocator and the performance of your application could be affected by heap fragmentation or even just the performance cost of heap allocations. A lot of C++ shops often wrote their own replacement containers that had a better allocation strategy. The design needs of a PlayStation 2 programmer ended up being different than an x86 desktop computer with virtual memory.

Well, this argument goes both ways, a generic implementation can never leverage special properties of your data. See i.e. Alexandrecu's talk [0] at about 1:05ff. If as a company you have special use cases or guidelines, you may easily benefit from libraries using these properties.

[0] https://youtu.be/FJJTYQYB1JQ

And then some dude somewhere deletes from repo his lib for left padding a string, and no one in the whole Facebook can compile anything until it's resolved? There's a reason why big companies like to have a full control over their code. Also it makes licensing a lot easier and legally safer.

C++ does have a standard package manager. It's the system package manager, instead of a language-specific one. Whether that's good or bad is a matter for some debate, but it does exist.

Sure, but you need to be root to install those, they're not portable, and there's no workspace isolation or build system consistency— yes CMake has bridge methods to work with autotools/pkg-config, but it's all pretty hacky.

For a tool that does attempt to do virtualenv-like workspaces for C++ packages, see colcon [1], but there are still a lot of rough edges there— it's an evolution of the catkin multi package build system/tool from the ROS ecosystem, and part of the promise of it is a pluggable mechanism for discovering the packages in your workspace to avoid needing to a patch in a special XML file containing dependency information.

[1]: https://colcon.readthedocs.io/

> C++ does have a standard package manager. It's the system package manager, instead of a language-specific one.

So C++ does not have a standard package manager.

Or are we really going full HN and debating the meaning of the word standard? Even if we are... is there a standard system? For that standard system, is there a standard package manager?

You're the only one trying to play with definitions of "standard" here. The system package manager was the standard packaging method for extra functionality on Linux (and other) systems regardless of implementation language before perl/Python/etc. started coming up with their own. You might argue that it's a bad standard, that it doesn't work for more "nimble" languages, whatever, but it is how C++ code is supposed to be packaged and has been for a long time. What C++ doesn't have is an exclusive package manager that can't be used with other languages. Non-exclusivity doesn't make anything less standard. Quite the contrary.

Maybe it's because the language give you all the tools to do it without any performance penalty. This allow you to have a custom library that is fit to your needs instead of a standard library that is globally good but obviously not optimal in all situations.

It may not be "standard", but Conan is a C++ package manager which IMHO is quite nice to work with. Not affiliated in any way.

Rust has a package manager, and yet I predict that the same thing is going to happen as the standard library is pretty thin and lacking. The reason is trust, you can't trust random third party dependencies if you're writing sensitive code, it's easier to trust a std lib OR something that pretends to replace/add to it from serious people.

Why do you think Rust would be different from Java or C#. I believe that both Java and C# are doing a good job at sharing package using Maven and Nuget.

I think another possibility might be to continually "roll" new wheels into the standard library.

python question: Do people re-implement a lot of the core library routines at every company? Compared to other scripting languages, I found python to have a pretty large/compatible standard library (except when they renamed the url/configparser module names)

Python is generally not used for the types of hard problems where a company specific library would help. When performance or code size matters you leave python for something else. Thus C++ will get libraries anytime the standard library had to make a tradeoff that isn't always right. Python by making the tradeoff for features has limited itself out of places where you would trade less features for something else. This is great for python as it wouldn't play well in the less features space anyway.

I expect to see company specific libraries for rust at some point because rust can play in the same space as c++ where you might want to make a different trade off.

I think this is a one sided view of Python, even though its true that it's most often used for easy problems. Eg Dropbox and Youtube are using Python, solve hard probs, probably have proprietary libs, and care about code speed.

thats why Rust might win in the long run because it have a common build system and package manager

I get to see a lot of folly code, since I'm at Facebook. It's mostly improved implementations and semi-obvious extensions of things that already exist in the C++ standard library, plus prototypes of things that will be in the next standard. Much like boost. While this is mostly good, it has to be said that improved implementations of bad patterns from the C++ standard (especially around concurrent programming) aren't something I'd spend my own time on. Life's too short to invent a better square wheel.

I don't want to end up splitting hairs over terminology so please correct me if I'm wrong. I would like expect a c++ library (folly or the standard library) to provide concurrency primitives and not patterns. Once you start providing patterns you are wading into framework territory. Is that the case with folly?

Unfortunately yes. The whole way that futures and "EventBase" are tangled up makes them more of a framework (and not one I would use if I had a choice) than a set of primitives.

https://github.com/facebook/folly/tree/master/folly/io/async

Thanks! I was going to ask "Does folly have its own asyncio for e.g." :)!

The C++ standard has included threading primitives since 2011. Many libraries exist that create higher level of abstractions, i.e. patterns, on top of these primitives. But you can also just use the primitive directly.

I found the README is not particularly useful in informing what the library does, but it seems to be a big bag of data structures, algorithms, and performance utilities.

The overview is much more informative: https://github.com/facebook/folly/blob/master/folly/docs/Ove...

Its amazing how often READMEs fail to describe what a repo actually does!

I suspect that this information seems so obvious to the maintainers that it gets overlooked. But its frustrating when a repo looks like it might be what I need but all I can find are installation instructions.

How does Folly compare to Google’s Abseil?

https://abseil.io/

Both are quite similar at a high level. They provide generic utilities for improving synchronization, strings, random number generators, high performance containers, etc. Both are no stable ABI guaranteed, live at head encouraged.

At a more detailed level, I find Folly much more substantial than Abseil. For example, both provide high performance hash tables. Abseil also provides a BTree-based map, which Folly does not. But Folly provides concurrent skip lists, LRU evicted hash maps, a high performance MPMC queue, etc. And that's just talking about data structures. Folly also has asynchronous I/O tools, futures, reference-counted buffers for IO, and many other things outside the scope of Abseil.

Personally, I am loving the asynchronous I/O mechanisms in Folly. It feels more expressive and results in cleaner code than boost ASIO. Just a first impression though, I am relatively new to the library.

Folly has Boost as a dependency and do not repeat features that already present in there. Not a problem if your project already uses Boost. Abseil seems to be self-sufficient.

Even i wonder about this! Also is Abseil widely used in the industry or only at Google?

Abseil is used extensively in Envoy Proxy. The project is a top level Cloud Native Computing Foundation (CNCF) project however Google are one of the top contributors.

I doubt it. Google lives at head and breaks things. Industry likes stability in general I have code from 1996 that I still need to support on cpus that were current back then. Nobody has written a new compiler for those cpus so I am stuck with whatever the old compiler supports.

OT: wiineeth, your account is shadow banned. I looked through your history and can't really tell why, but you should email hn and ask them to lift the ban, since IMO you don't look like a bad member in a community.

We just started using it for its flat_hash_map.

There's a pretty good CppCon talk about how they made std::string suck less at facebook

https://youtu.be/kPR8h4-qZdk

Does anyone know what happened to the Facebook https://opencompute.org project? Does anyone actually use that? Seemed interesting at the time but haven't heard much recently.

I would've thought there's a better way of approaching software design in 2020 than "This 50,000 employee company decided to open source one big bang grab bag of functions". Especially with a readme that doesn't include any explanation of what's in the library or why they're there.

>Folly (acronymed loosely after Facebook Open Source Library)

You literally broke your first promise. That literally translates ot FOSL. You could've used FOSL. That's a word. It makes sense. It's even a bit funny.

>is a library of C++14 components designed with practicality and efficiency in mind.

As opposed to all those components that are designed with impracticality and inefficiency in mind. I honestly don't think I can say I've ever designed anything without those two things in mind.

>Folly contains a variety of core library components used extensively at Facebook.

Which tells me literally nothing about what's in this library. How do I know what's used extensively at facbook? Last I saw it was mediocre PHP 4.

Perhaps you missed the "What's in it" section while skimming the readme? :)

>Because of folly's fairly flat structure, the best way to see what's in it is to look at the headers in top level folly/ directory. You can also check the docs folder for documentation, starting with the overview.

Ah, the old "Read the code" version of documentation. Guess what? That's not documentation. That's douchebag for "I didn't write documentation"

That doesn't seem very fair. They link a docs folder and an overview document, neither of which require that the user read code to get a good idea of the library's contents.

That said, the code is also reasonably well commented as well, for example: https://github.com/facebook/folly/blob/master/folly/Singleto.... So "read the code" is not terrible advice in this case either.

I am quiet impressed by Facebook R&D and whole technology stack they use.

A thread from 2012: https://news.ycombinator.com/item?id=4059356

Sad to see that almost all companies have to implement their own string because of the difficult to use and slow performance std::string.

thank you facebook

The fact that the master branch CI has a high rate of failure doesn't inspire confidence in the development process.

I assume your looking at the runs in GitHub actions?

The development process itself is fine. This takes place inside Facebook on their own infrastructure. The problem is their process to automatically sync their internal repo to GitHub and keep the public CI green is broken.

What a folly!

"folly requires gcc" That's surprising to me. Clang and the associated tools are so useful I would have expected all tech companies to have migrated to it.

I'm pretty sure that "folly requires gcc 5.1+" is meant to be a statement about the minimum GCC version, and not that GCC is the only compiler that will work.

If you poke around the github issues, you'll note that several folks are successfully compiling folly with Clang.

Don't know why people downvoted you. The intended interpretation was "if gcc, must be 5.1+", but I can see how it might be misunderstood as it was written.

The readme was updated and now reads

> folly supports gcc (5.1+), clang, or MSVC. It should run on Linux (x86-32, x86-64, and ARM), iOS, macOS, and Windows (x86-64). The CMake build is only tested on some of these platforms; at a minimum, we aim to support macOS and Linux (on the latest Ubuntu LTS release or newer.)

folly is used in FB opensource projects on Windows, macOS, and Linux (e.g. watchman) and Android and iOS (React Native, though that might not include the whole library.)

The highlighted sentence gives the minimum version of gcc which folly broadly supports.

One motivation for this is so that a user may know whether folly supports the version of the gcc c++ compiler which is preinstalled on their system, since many systems ship with a preinstalled gcc. Likewise for systems which ship with a blessed package repo.

But folly also supports recent versions of clang and msvc, as other commenters here note.

It seems like a pretty mature cross-platform library.

It could be their statement "folly requires gcc 5.1+" meant gcc 5.1 or greater, where clang might be greater?

> It could be their statement "folly requires gcc 5.1+" meant gcc 5.1 or greater, where clang might be greater?

Reminds me of an old joke: "The package said it required Windows 95 or better, so I installed Linux."

They'd save a lot of guessing by putting in which Special Modern Features their code relies on.

> They'd save a lot of guessing by putting in which Special Modern Features their code relies on.

The first paragraph of the readme starts with "Folly (acronymed loosely after Facebook Open Source Library) is a library of C++14 components" and gcc 5.1 is the first version (https://gcc.gnu.org/projects/cxx-status.html#cxx14) which supports all of c++14

Could you elaborate on that? What aspects of using clang as opposed to gcc were so beneficial?

The sanitisers are a good reason (UBSan, ASan). Clang compilation is faster than GCC, also.

Don't know if it's still true, but when I was at Facebook we used Clang for debug builds, and GCC for production builds.

Is it just me or is that a recipe for Heisenbugs?

Better error messages

I'm talking about the sanitizers, clang-tidy, clang-format, and of course the error messages of clang itself.

I could be wrong, but, a quick Googling / wiki shows clang is from 2007, whilst facebook was founded 2004, and even though clang was available in 2007, it takes years before people consider using newer tools like clang over proven technology, especially in the C space.

I posted this elsewhere in this thread — Don't know if it's still true, but when I was at Facebook we used Clang for debug builds, and GCC for production builds.

Google is primarily using clang, not gcc anymore.

Reply on news.ycombinator.com