Caffe2 adds 16 bit floating point training support on the NVIDIA Volta platform | Hacker News Reader

Caffe2 adds 16 bit floating point training support on the NVIDIA Volta platform

caffe2.ai

138 pointsbwasti51 comments

From first looks, there is little doubt that that NVIDIA's Volta architecture is a monster and will revolutionize the AI and HPC market. But the article seems to avoid quantifying how 16-point FP operations are beneficial against 32- or 64-bit FP operations in real-world usecases, or how the Caffe2 / NVIDIA architecture provides any significant boost to FP16 in particular, especially apropos to images (or why FP16 is better for image data in general).

I'm interested more in understanding why Caffe2 would outperform Theano, Tensorflow, MXNet, etc. once Volta chipsets are generally available, beyond early pre-release optimization, particularly when most of the front-runners are already leveraging / taking into account NCCL, CuDNN, NVLink, etc. When the burden of adding support for new NVIDIA primitives is so low, what gives Caffe2 an advantage beyond an ephemeral "we were partners with NVIDIA first" one-up that would last for a couple of months at most?

(Apologies in advance if this post sounds overly negative, but I am constantly evaluating the current crop of frameworks for the trade-offs they enforce on the problem space, and a definitive answer would be very helpful.)

Caffe2 seems to be engineered for efficiency as opposed to flexibility. For example, Caffe2 is apparently Facebook's go-to choice for mobile edge device deployment for applications such as style transfer.

Efficient how ? In terms of memory management ?

TPUs are smaller in terms of Si real estate and burn less power. Also, they can be made much faster.

I mean, it uses a static graph definition instead of dynamic graphs (allows for deployment optimizations), but is there anything Caffe2-specific that would give it a leg up over Theano/TensorFlow/whatever other static graph library?

16fp is faster, and the increased precision is not needed in most cases

(In fact there are even some experiments with a very low number of bits - can't find the link now though)

16 bit FP operations seem to be more than enough for most networks - in fact for inference, google claims in their TPU paper that most network parameters can be quantized down to 8 integer bits!

There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.

>>> I'm interested more in understanding why Caffe2 would outperform [...]

A vendor-made benchmark where the vendor is outperforming other vendors.

Let's not jump to conclusion on what's really fastest.

16 bits lets them build something that goes much faster and/or faster per watt and neural nets don't need even 16bits usually. pretty simple!

On the topic of FP-

https://dennisforbes.ca/index.php/2017/04/11/floating-point-...

Most desktop chips offer little advantage for FP16 (at best that the source memory footprint is smaller, maybe better cache hit rates), but newer GPUs can actually blaze. If it has the magnitude/precision needs for a task, it's a big win.

FP16 multipliers are smaller (in terms of silicon footprint), which means that you can fit more of them on a given silicon die, and they use less power than FP32. The other advantage is that FP16 uses half as much memory, meaning that FP16 operations require half as much memory traffic, which can lead to faster computations. It can also allow you to fit models on your GPU which have twice as many parameters.

The reason you don't typically have FP16 on CPUs is that in the CPU world, cache dominates die area and power usage. The floating point ALU component of a CPU is relatively tiny. Still, even in the CPU world, Intel recently added instructions to convert to/from FP16 before operating on it, because they know there is a memory bandwidth advantage in using a smaller floating-point format.

The AnandTech article on the Volta has a lot more information on the new architecture: http://www.anandtech.com/show/11367/nvidia-volta-unveiled-gv...

It's interesting the speed up isn't more pronounced between Volta and Pascal considering the Tensor cores on paper give you about 6x the MFlops. The price differential looks large.

From AnandTech: "By the numbers, Tesla V100 is slated to provide 15 TFLOPS of FP32 performance, 30 TFLOPS FP16, 7.5 TFLOPS FP64, and a whopping 120 TFLOPS of dedicated Tensor operations. With a peak clockspeed of 1455MHz, this marks a 42% increase in theoretical FLOPS for the CUDA cores at all size. Whereas coming from Pascal, for Tensor operations the gains will be closer to 6-12x, depending on the operation precision."

Are numbers available for FP16 on P100 or FP32 on V100? It would make for a more direct comparison.

EDIT: Nvidia's advertised TFLOPS are:

         FP16  FP32  FP64
    V100 30    15    8.5
    P100 21.2  10.6  5.3
    K40  4.29  4.29  1.43

Your table doesn't include the new "tensor core" TFLOPS of V100.

That's a core that does 4x4 FP16 matrix multiplication + 4x4 FP32 accumulation in one go.

That's where V100 gets its boost, up to 120 TFLOPS.

Tensor Cores: 120 TFLOP/s mixed-precision (peak). Typo in your table: V100 FP64 is 7.5 TFLOP/s.

The GPU looks like a monster, and I am sure it'll deliver more power to AI applications, but what I really want to know is when I can put one in my gaming PC.

I think it's great that what started as a specialist gaming device is now being used in industry for Big Things. The development cost that Nvidia et al. have invested in new designs has undoubtedly been financed in (large) part by the gaming community. Now income and advancements for both sectors feed into the other and gamers like me are reaping the benefits with reduced price:performance across the range.

Meanwhile I still remember being seated at the Games Development Conference talk in 2009 where Intel tried to convince us how Larrabee would change the world of computing.

Still waiting for them to produce anything worthwhile buying instead of AMD and NVidia GPUs.

Actually, it's now called Xeon Phi: https://en.wikipedia.org/wiki/Xeon_Phi

I know, they are pretty hard to get and as such largely ignored by HPC and gamming communities.

Their main sales pitch is "given that it's a bunch of x86 put together, you don't have to port your code to get Massive Paralellization by Intel (TM)". Some supercomputers use them, but it's true: I've never seen a worthwhile comparisson between that and a strong (or equivalent) GPGPU.

There are some very interesting talks by Intels experts on how to use these guys with AVX and AVX512 (or however its called now) and the multiplicative improvement both multicore and AVX give when used together (the sales line goes along like this "10x for multicore, and on top of that 4x for AVX, voilá 40x speedup!").

I don't work in an HPC environment, so I don't really know if they stand up in reality to Intels claims.

Hard to get? For better or worse TTBOMK there are more Intel Xeon Phi machines on the TOP500 than GPU accelerated systems.

Yes for people with normal bank accounts vs random computer store.

These efforts are mostly going to aid further market segmentation and boosting profits; e.g. even for development you'll now need a Tesla/Quadro that costs a ton of money, the days of GTX cards that have the same GK2xx or GM2xx chip as Teslas is over.

While the "tensor" units may land in consumer stuff (though I don't think it'll happen this year) I expect there to be little "trickle-down" planned or wanted at this stage.

I'm a long time Nvidia user. However, the recent article headlines have been a word salad.

The new tesla volta super flip flop at 1.21 gigawatts blah blah blah.

Just saying.

Also Kudos to Nvidia for the buzzword/made up word creation for their products.

Surely getting people to refer to their vector lanes as "cores" was their biggest piece of marketing magic. Yes, they're more flexible than SIMD cores but not more so than an OoO core's execution ports.

I can see why they launched 1080 Ti early. Had I not seen this I'd definitely not be waiting for Volta.

How are things in the red camp ? There was some HIP thing where Fiji was as good as Pascal.

Any idea on the timeline for consumer Volta?

Early 2018 depending on GDDR6 supply. Seems like Hynix will start mass production on in late 2017.

Phoronix said "Volta desktop graphics cards are expected to succeed Pascal in late 2017 or early 2018" in https://phoronix.com/scan.php?page=news_item&px=NVIDIA-Volta....

German news portal heise.de said Q2 2018.

The GTX 1070 for laptops is destined to be the most interesting price/performance opportunity for a while imho, you can even undervolt the CPU -0.100 to -0.150 to reduce overheating.

I was told Q4 2017 for non-graphics consumer (what kind of consumer are you referring to), but that is earlier than most others are saying. https://news.ycombinator.com/item?id=14310633

Still no good CuDNN equivalent from AMD for machine learning that would make their cards competitive for that use case

Waiting for Vega, which is AMD's answer to the 1080

it seems increasingly common that that's the team red mantra nowadays, "Wait for x".

Wait for ryzen.

Wait for vega.

Wait for ryzen again.

Amd cpu line is rolling out. You can buy ryzen.

Vega is coming out this year. Volta isn't consumer based. Go check the enterprise prices.

Consumer Volta is mid 2018, so this, from a consumer perspective, is a paper launch.

That's a blanket-statement and ends up being incorrect/inaccurate. The top Vega card is HBM2, not even in the same ballpark as the 1080.

Also, recent Linux patches hint that there will be dual-chip water-cooled Vega boards too which is likely not even 1080 Ti regime: https://www.techpowerup.com/233208/linux-drivers-point-to-up...

In consumer terms AMD's got a compelling product for mid-range GPUs where most of the money is but NVidia's got the top of the range locked up. Hopefully we'll see some good chips out of AMD soon which combine a genuinely good CPU core with a good GPU in one package which'll be really nice in the low end of the market.

As for scientific computing I suppose there are workloads where having your GPU and CPU on one die might give you latency savings that are worth it but mostly NVidia is going to keep owning that market for the next few years.

The new "Radeon Instinct" cards are about to be released: http://www.anandtech.com/show/10905/amd-announces-radeon-ins...

The mid-range has a lot of potential as it's based on Fiji which NVIDIA can't match in terms of memory BW with any of their existing ML products (nor near future ones). This is important IMO because ML inference is memory-bound and GDDR5 on NVIDIA's cards is no match for even 1st gen HBM.

The big cards is announced to be 12.5/25 Tflops SP/HP, so as long as it's not "tensoring" it should be in between P100 and V100. If they get the price right (and given that even P100 is currently still $5.5-6 w/o tax), it will hopefully gain traction.

A lot hinges on their fully OSS (!) software stack, but they are getting things done, just had a new release: https://rocm.github.io

I understand the pedigree of these cards, but at what point do we stop calling them GPUs and start calling them something else? MPU? TPU? I don't know, but isn't it a little bit weird to keep using the word "graphics" for something that is being made more and more specifically for other things?

I don't see the point of adding to the confusion. They've been called GPUs for ever, everybody agrees on the term, why not continue to use it?

I like the term "GPGPU" (General-Purpose Graphics Processing Unit) as it's at least explicit about its general-usefulness.

Eh, that seems like it’s intended for general purpose graphics (i.e. 2D business applications vs “special purpose” 3D, gaming, VR, etc.)

No. For many years now, gpgpu has meant 'using gpu's for non-graphics calculations'.

Understood. I’m just commenting on the name in a vacuum. If it’s supposed to mean use for non-graphics calculations, it’s confusing to have “graphics” in the name.

It's the exact opposite of general purpose though, it's several specific purposes.

No, it's several specific purposes that heavily use the provided general primitives. There's nothing stopping more things from using it, which makes it general.

You don't call your PC a specific purpose computer just because all you do is use google chrome.

Nvidia rep: "Our latest MPU will train your models lots faster".

Potential customer: "What's an MPU?"

N: "Oh, it's the same thing we used to call a GPU"

P: "Why did you change the name?"

N: "Because it's a little bit weird to keep using the word 'graphics' for something that is being made more and more specifically for other things"

P (puzzled, and marginally less likely to make a purchase): "Oh right"

You can still call it a GPU, but that now stands for General Processing Unit. It pairs well with CPU, being the Central Processing Unit in the center of the entire machine and GPUs being added for additional processing capability.

Can the Volta architecture be used to run WebGL in a Browser?

cudnn 7?

Reply on news.ycombinator.com