Comment by maldeh | Hacker News Reader

Comment by maldeh | Hacker News Reader

Parent Full thread

From first looks, there is little doubt that that NVIDIA's Volta architecture is a monster and will revolutionize the AI and HPC market. But the article seems to avoid quantifying how 16-point FP operations are beneficial against 32- or 64-bit FP operations in real-world usecases, or how the Caffe2 / NVIDIA architecture provides any significant boost to FP16 in particular, especially apropos to images (or why FP16 is better for image data in general).

I'm interested more in understanding why Caffe2 would outperform Theano, Tensorflow, MXNet, etc. once Volta chipsets are generally available, beyond early pre-release optimization, particularly when most of the front-runners are already leveraging / taking into account NCCL, CuDNN, NVLink, etc. When the burden of adding support for new NVIDIA primitives is so low, what gives Caffe2 an advantage beyond an ephemeral "we were partners with NVIDIA first" one-up that would last for a couple of months at most?

(Apologies in advance if this post sounds overly negative, but I am constantly evaluating the current crop of frameworks for the trade-offs they enforce on the problem space, and a definitive answer would be very helpful.)

16 bit FP operations seem to be more than enough for most networks - in fact for inference, google claims in their TPU paper that most network parameters can be quantized down to 8 integer bits!

There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.

>>> I'm interested more in understanding why Caffe2 would outperform [...]

A vendor-made benchmark where the vendor is outperforming other vendors.

Let's not jump to conclusion on what's really fastest.

FP16 multipliers are smaller (in terms of silicon footprint), which means that you can fit more of them on a given silicon die, and they use less power than FP32. The other advantage is that FP16 uses half as much memory, meaning that FP16 operations require half as much memory traffic, which can lead to faster computations. It can also allow you to fit models on your GPU which have twice as many parameters.

The reason you don't typically have FP16 on CPUs is that in the CPU world, cache dominates die area and power usage. The floating point ALU component of a CPU is relatively tiny. Still, even in the CPU world, Intel recently added instructions to convert to/from FP16 before operating on it, because they know there is a memory bandwidth advantage in using a smaller floating-point format.

Caffe2 seems to be engineered for efficiency as opposed to flexibility. For example, Caffe2 is apparently Facebook's go-to choice for mobile edge device deployment for applications such as style transfer.

I mean, it uses a static graph definition instead of dynamic graphs (allows for deployment optimizations), but is there anything Caffe2-specific that would give it a leg up over Theano/TensorFlow/whatever other static graph library?

Efficient how ? In terms of memory management ?

TPUs are smaller in terms of Si real estate and burn less power. Also, they can be made much faster.

On the topic of FP-

https://dennisforbes.ca/index.php/2017/04/11/floating-point-...

Most desktop chips offer little advantage for FP16 (at best that the source memory footprint is smaller, maybe better cache hit rates), but newer GPUs can actually blaze. If it has the magnitude/precision needs for a task, it's a big win.

16fp is faster, and the increased precision is not needed in most cases

(In fact there are even some experiments with a very low number of bits - can't find the link now though)

16 bits lets them build something that goes much faster and/or faster per watt and neural nets don't need even 16bits usually. pretty simple!

Reply on news.ycombinator.com