NNVM Compiler: A New Open End-To-End Compiler for AI Frameworks

Up until recently at work I had to work with Keras on CPU-only and the easiest way to get things working was TensorFlow backend. Earlier this year the 1.3 update gave me roughly 15% performance increase just from

    pip install tensorflow --upgrade

Fortunately I got a new machine with a cuDNN capable GPU, but I'll be testing the NNVM for Keras backend when its working. We might get to squeeze some epochs out of my old machine, now in the hands of a secretary who won't really notice we're training a net while he replies some emails.

part of TVM https://github.com/dmlc/tvm is built with primitives in Halide. Halide is indeed super cool and TVM benefit a lot from its experience. While Halide optimizes CPU and image processing workload well. TVM also focus specifically on deep learning and offers more optimizations on multi-core, GPU and other hardwares. This requires rethink of quite a lot of designs.

NNVM compiler is built on top of TVM, with additional graph level optimizers, and the two forms an end to end pipeline

Calling it a fork isn't really fair. It just reuses bits and pieces from Halide where it made sense for them to do so, and it's appropriately credited. We're happy for people to build on our stuff.

- One of the Halide people.

My (limited) understanding is that 4-8 GPUs will be around 2 orders of magnitude faster than a CPU at training, and that it's not very realistic to use a CPU for training anything but the smallest models.

It's slower but can be cheaper for cloud training. See http://minimaxir.com/2017/07/cpu-or-gpu/

Those long training times are for training from scratch. Most image tasks don't need that, although text tasks often do.

> I keep hearing about outrageous training times for GPUs on the order of weeks with something like 4-8 GPUs

This is for training a competitive model from scratch on a fundamental problem like image recognition. If you don't care about the last 1-2%, it's possible to train a useful model in a few hours (but still on a GPU).

> is anyone actually using CPUs instead

There are useful things you can do without a GPU. For example, "Transfer learning", which can be as simple as chopping off the last layer of someone else's GPU-trained model and substituting your own, can be done on a CPU in reasonable time. This is because you typically need less data and because far fewer parameters need to be fit.

For example, HBO's "Hotdog/Not-hotdog" app was done this way. See their description of "V1" here: https://medium.com/@timanglade/how-hbos-silicon-valley-built...

I wouldn't recommend for training. CPU's work just fine for low intensity inference. Especially online/interactive inference, these are situations where you pass in one example at a time. GPU excel when you pass data in continuous predictable batches.

Well... I feel like people are only focusing on training. CPUs for inference are sometimes helpful in online inference settings where you don't ever really "batch" requests. In this case, they can be cheap.