simonmar

Sorry about that, the package was still set to private. Try again now?

We want to open-source the C++ and Python indexers but they're not ready yet - we have to separate them from internal build-system-specific bits.

Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/s...

For wiring up the indexer, there are various methods, it tends to depend very much on the language and the build system. For Flow for example, Glean output is just built into the typechecker, you just run it with some flags to spit out the Glean data. For C++, you need to get the compiler flags from the build system to pass to the Clang frontend. For Java the indexer is a compiler plugin; for Python it's built on libCST. Some indexers send their data directly to a Glean server, others generate files of JSON that get sent using a separate command-line tool.

References use different methods depending on the language. For Flow for example there is a fact for an import that matches up with a fact for the export in the other file. For C++ there are facts that connect declarations with definitions, and references with declarations.

I should point out that Glean has evolved quite a bit since that talk!

There will be more indexers: we have Python, C++/Objective C, Rust, Java and Haskell. It's just a case of getting them ready to open source. You can see the schemas for most of these already in the repo: https://github.com/facebookincubator/Glean/tree/main/glean/s...

The graph is sorted by performance, with the worst performing (not necessarily the most common) on the left. We've also done more profiling and optimisation since we took those measurements.

FXL employed some tricks that were sometimes beneficial, but often weren't - for example it memoized much more aggressively than we do in Haskell. Mostly that's a loss, but just occasionally it's a win. When a profile shows up one of these cases, we can squash it by fixing the original code.

What matters most is overall throughput for the typical workload, and we win comfortably there.

The memo table (map) is a bit of state that is maintained throughout the request's lifetime. When we compute a memoized value, it is inserted into the map, and if we need the value again we can just grab it from the map instead of recomputing it.

The "automatic" bit is that we insert the code that consults the map so the programmer doesn't have to write it. The map itself is already invisible, because it's inside the monad. So the overall effect is a form of automatic memoization.

For example, let's say that one of the things you want to compute is the number of friends of the current user. This value is used all over the codebase, but it only makes sense in the context of the current request (because every request has a different idea of "the current user"). So this is a memoized value, even though in the language it looks like a top-level expression.

Memoization only stores results during a request. It starts empty at the beginning of the request and is discarded at the end, and it is not shared with any other requests. It's just a map that's passed around (inside the monad) during a request.

Here's our paper about the ideas behind Haxl: http://community.haskell.org/~simonmar/papers/haxl-icfp14.pd...

He is still committing, but not quite so often :)

Haxl is the only major project at FB using Haskell right now, but who knows where this will lead?

Segmented stacks are better even if you have a relocatable stack. GHC switched from monolithic copy-the-whole-thing-to-grow-it to segmented stacks a while ago, and it was a big win. Not just because we waste less space, but because the GC knows when individual stack segments are dirty so that it doesn't need to traverse the whole stack. To avoid the thrashing issue we copy the top 1KB of the previous stack chunk into the new stack chunk when we allocate a new chunk.

Oh yes, immutability is crucial. Generational GC already makes you pay for mutation with a write barrier, and in our parallel GC we omit the locking when copying immutable objects, accepting that a few might get copied twice. In the new GC mutable objects become even more expensive. I don't think local GC is viable at all in a language with ubiquitous mutation.

We still have lots of tuning to do, I'll be looking in detail at what is going on during those pauses. This program is slightly atypical in doing quite a lot of old-generation collections though (you can see two in the graph), it has a low infant mortality and might benefit from a larger nursery size. (don't you love GC analogies? :)

Yes, the GHC design has certainly been influenced by Manticore (that was one of the "other designs" I referred to). Though in GHC we do have some different problems to solve, the worst of which is that we have to support a bunch of programming abstractions that use mutation.

Yes, a fair point. Concurrent and parallel do not describe disjoint sets of programs, of course. However, they really are different concepts. I think wikipedia's page on concurrency could use some work to make this clearer, maybe I'll try to gather enough references to make the case.

If you want to rephrase the title, perhaps "Parallelism =/> Concurrency" would be better ("=/>" is "does not imply").