Flashlight: Fast and flexible machine learning in C++

From my understanding, Tensor Comprehensions and Halide are both very tentative research projects.

> not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow).

I don't think this is an accurate recreation of the history that led to FB working on pytorch.

As far as Facebook engineering resources are concerned, I believe facebook takes the approach that supporting open source high visibility projects is itself a recruiting draw for pulling talented engineers into their umbrella.

I don't see a contradiction here, in fact it makes sense.

You have some goals, and a couple of promising approaches. It's hard to say which will work better, but there's enough budget and people to work on them to just try them both and see.

I've heard anecdotes of similar strategies at banks, who have sufficient budget to hire two parallel teams to build literally the same product, sometimes without even knowing about each other. At the end, the one that ends up being faster/better gets used.

I guess it's like a microcosm of free market competition within an org, as opposed to top down planning.

FB makes $1.3M/employee while average pay is $120K, i.e. less than 10% - thus the employees are actually very cheap and FB can waste a lot of employee resource, like doing 10 ML frameworks, and even though ML related employees earn $1M+ it still wouldn't move the needle financially for FB.

> Flashlight’s modular internals make it a powerful research framework for research frameworks.

> We’re already using Flashlight at Facebook in our research focused on developing a fast speech recognition pipeline, a threaded and customizable train-time relabeling pipeline for iterative pseudo-labeling, and a differentiable beam search decoder.

> Our ongoing research is further accelerated by the ability to integrate external platform APIs for new hardware or compiler toolchains and achieve instant interoperability with the rest of Flashlight.

They found it easier and faster to make a framework optimized for their research than to try to iterate on a larger and more complex codebase. I think there is less cognitive overhead and less things to change when experimenting with a new idea, and since they've optimized for fast rebuild times, it is much faster to try out new things. They get native integration with the C++ ecosystem for free and since their team all know C++ well, it makes sense to just do it in plain C++.

A lot of it is also asynchronous for performance: the Python code just enqueues more work to a queue which some native C++ code processes. For TensorFlow the Python code traces an entire computation graph that is stored a protobuf and then executed by a C++ native stack, potentially remotely/distributed. Serving ML with TensorFlow does not involve any Python code in many scenarios.

Python is still quite useful for scientist to quickly glue everything together, and to describe their dataset, or when they collect result and need to produce graphs or other data analyses.

Most quants do not want to learn complex build systems that have quirky behavior on different platforms, wait for very long compile times when making small changes to the code, dense and incomprehensible error messages, and a host of painful problems that one has to consider when writing C++.

Python just works really well on almost all platforms.

The biggest downside of Python is its parallelism, which means there is a lot of hackyness around writing parallel code. In most cases we can break things down and run different tests independently of one another, but in many other cases we have to make use of awkward workarounds, use multiprocessing, and other tricks.

C++ seems ideal for me right now because it is the only other language with a somewhat mature stack (perhaps Julia as well, but I haven't played too much around with that).

Julia has a fast maturing data wrangling super-project (Queryverse).

- Python allows for higher level description of algorithms, which means researchers can focus more on the ML stuff and less on low level details.

- There is no performance gain in going from Python to C++, because in both cases the models are compiled to specific binary formats to be executed on dedicated hardwares. TensorFlow enables accelerators not only for training, but also for data transformations and preprocessing.

(Although not, let's be honest, without amusement.)

I assume they are going for "batteries included" connections?

that said, interesting decision to ditch the scripting language. when i think of these sorts of things, immediately think that embedding or supporting a scripting language makes a ton of sense. i'm curious what the thinking was to just go full c++? i suppose they just decided that modern c++ was easy enough and this was better for simplifying production?

Perhaps because I can't see very well I had uncertainty when reading the article title. But unless you're quite innocent (i.e. not a user of the internet) I don't see how you could avoid knowing about such devices. Knowing many programmers, I also know that they span the full spectrum from Bible-studies to BDSM (though that might be a horseshoe), and many are prone to juvenile humour which can become sexually toned. So I know that someone will use it in this context. When you become a manager you choose to avoid creating such problems. Some of you probably see this tone policing -- you are correct, suck it up.

For those who don't understand what Hamming distance is: https://en.m.wikipedia.org/wiki/Hamming_distance