Hacker News Reader

Comment by spydum | Hacker News Reader

Curious to hear what exactly the devices do? Article hints at compression and such, and the fact that they've moved devices to the edge of the machine to handle network connectivity makes me think it's a shift for better interaction with SDN more than anything else. I don't get how it has much to do with AI though?

View on HN

In Bing, it's used to score results for queries: "One of these FPGA nodes did the Bing search scoring for a 48-node server pod, and was linked to the nodes by four 10 Gb/sec Ethernet ports." http://www.enterprisetech.com/2014/09/03/microsoft-using-fpg...

Here's a paper about it: https://www.microsoft.com/en-us/research/wp-content/uploads/...

4.4, Feature Extraction: "The first stage of the scoring acceleration pipeline, Feature Extraction (FE), calculates numeric scores for a variety of “features” based on the query and document combination. There are potentially thousands of unique features calculated for each document, as each feature calculation produces a result for every stream in the request—furthermore, some features produce a result per query term as well. Our FPGA accelerator offers a significant advantage over software because each of the feature extraction engines can run in parallel, working on the same input stream. This is effectively a form of Multiple Instruction Single Data (MISD) computation."

FTA

> ... in the coming weeks, they will drive new search algorithms based on deep neural networks—artificial intelligence modeled on the structure of the human brain—executing this AI several orders of magnitude faster than ordinary chips could.

That raises as many questions as it answers. Why FPGAs and not GPUs which can run just about any deep neural network but usually faster and more efficiently?

One reason is FPGAs are more flexible.

Sure, GPUs deliver impressive raw performance. To be useful, the task must benefit from massively parallel hardware. GPU hardware works fantastic for shading polygons, training neural networks, or raytracing. For compression and encryption algorithms however, GPUs aren’t terribly good.

Another reason is while a GPU delivers impressive bandwidth on parallel-friendly workloads, it’s usually possible to achieve lower latencies with FPGA. An FPGA doesn’t decode any instructions, and its computing modules exchange data directly.

GPUs give you great throughput, but they're expensive, eat a lot of power, only work well in batch and aren't tuned to prediction (eg poor INT8 performance and too much VRAM).

GPUs have worse performance per watt than a tuned FPGA. Some newer FPGAs can have 400 megabits of on chip RAM - that's huge, significantly larger than the 128-256k cache typically available on chip for a GPU that turns into big energy savings.

GPUs have worse performance per watt than a tuned FPGA

Citation needed.

Maxwell Jetson TX1 is claimed to achieve 1TFlops FP16 at <10W, and soon to be released Pascal based replacement will probably be even more efficient.

While I dont have any external publications addressing this general claim, this is taken from my current and past experiences with internal studies focused on neural networks implemented on the TX1, other GPU, custom ASICs, and FPGA approaches. In terms of power efficiency it generally goes ASIC > FPGA > GPU > CPU. If you're doing just fp32 BLAS it's hard to beat a GPU, but it turns out many problems have features that you can optimize for.

The TX1 power consumption including DRAM and other subsystems peaks 20-30W. Typical usage is 10-15W if you're running anything useful.

That 1 TFLOP counts a FMA instruction as 2 flops - while accurate and useful for say dot products - for other workloads the throughput will be half of this number.

As an example of an FPGA performing significantly better than the TX1 is DeepPhi [0].

[0] http://www.deephi.com/en/technology/

In that link, where's the comparison of fpga vs TX1?

If you click on Papers, there is a link to "Going Deeper with Embedded FPGA for Convolutional Neural Network", which compares against the TK1: https://nicsefc.ee.tsinghua.edu.cn/media/publications/2016/F...

While not the TX1 vs FPGA result you want, this is very close. For example they aren't using the latest FPGA or GPU, and are not using TensorRT on the GPU and on the FPGA side they are using fatty 16-bit weights on an older FPGA rather than newer stuff you can do with lower precision (which improves the efficiency of the FPGA having more high speed RAM collocated with computation vs GPU which is primarily off-chip).

If you want to learn more about this stuff, I suggest a presentation by one of Bill Dally's students (chief scientist at NVIDIA): http://on-demand.gputechconf.com/gtc/2016/presentation/s6561...

Thanks, but TK1 is using FP32 weights, as opposed to FP16 on FPGA. If you double the GOP/s number for TK1 to account for that, you will end up with pretty much identical performance, and the paper claims they both consume ~9W.

I'm not saying you're wrong, just that to make a convincing claim that FPGAs are more power efficient than GPUs, one needs to do an apples to apples comparison.

And of course, let's not forget about price: Zynq ZC706 board is what, over $6k? And Jetson TK1 was what when released, $300? If you need to deploy a thousand of these chips in your datacenter, to save a million per year on power, you will need several years to break even, and by that time, you will probably need to upgrade.

It just seems that GPUs are a better deal currently, with or without looking at power efficiency.

Anything they want. I think it's like having a blank breadboard circuit, Microsoft can mock up a new purpose designed chip and program it in to these FPGAs instead of having to fab one up from scratch.

It's hard to tell from the article though, I'm just guessing.

There are resource limitations to contend with, roughly like an limit on total number of logic gates (e.g. AND, NOR,..) that can be "rewired", and also the bus speed to memory can become prohibitive for some applications. This is why GPUs have been so successful at the subset of problems they are targeted at, just like GPUs as compared to CPUs. However, FPGAs are becoming ever more malleable and heterogeneous: the embedded toolkit (like on-chip hardwired DSPs) is expanding, and the potential for live reconfiguration of parts of an FPGA while other parts are in-use (dynamic/active partial reconfiguration) [0] is pretty exciting.

Edit: Though I think it was not mentioned in the article, Microsoft and Intel/Altera have indeed gone this route in no small part due to the empirical death of Moore's Law (which has gone much discussed on HN over the past few years).

[0] https://en.wikipedia.org/wiki/Reconfigurable_computing#Parti...

Yeah, not thinking they are all powerful, just sounds like they can be fine tuned easier than designing a whole new for-purpose chips.

Thank you for the info though!

Reply on news.ycombinator.com