Facebook to open-source AI hardware design | Hacker News Reader

Facebook to open-source AI hardware design

code.facebook.com

197 pointskungfudoi59 comments

I'm glad they're open-sourcing this, but I have to say that making 8 GPUs work together is not that big of a deal. Companies like Cirrascale are making up to 16 GPUs scale linearly with a blade.

so if it's not the best, then it's not worth it?

We have distributed implementations that allow us to use more than 8 GPUs to train a single network. These machines have 8 cards each, but we aren't limited to a single machine (though having more GPUs connected via PCI than via Ethernet helps)

Any particular reason you don't have Infiniband on these for interconnects?

Having this chassis with Infiniband and a local disk would be a dream if the manufacturing cost is right as we scale up in our local datacenter where I'm at.

That's interesting. But from what I understand, the problem isn't getting many GPUs to train a single network -- the problem is to get them to scale linearly; i.e. to get each additional GPU to accelerate the process as much as the last. Throwing boxes at neural net training is easy, but people run into serious plateaus scaling GPUs.

It's irrelevant if you can't balance the cost, CPU, shared DRAM, and - possibly most importantly - PCIe and network connectivity in and out of it. A typical mid-range Intel Xeon has 40 PCIe lanes. A high-end GPU requires 16. You can do the math. Facebook has decided upon a balance that works for them and their workloads; it's quite likely that going to 16 GPUs in the chassis resulted in overall worse utilization because of PCIe bandwidth limits, or socket and QPI count, etc. What's a big deal is someone having gone and done a lot of work to calculate and experiment with finding a balanced design that's also cost-effective and easy to maintain.

Let me guess, this "balanced design" is 4x8 on one socket and 4x8 on the other.

Actually the PCI express topology is configurable, which is one of the innovations. You can put all 8 gpus on a single CPU bus or have 4 on each CPU (but have to use QPI between them)

Big Sur is twice as fast as our previous generation, which means we can train twice as fast and explore networks twice as large.

Can a machine learning expert comment on this claim? There are many domains in computing where doubling your compute power does not halve your execution time, or allow you to double the size of your input. (For example, your algorithms are worse than O(n), or you have hardware communication bottlenecks.)

These are machine learning experts making the claim?

Machine learning is usually embarrassingly parallel.

If I was reviewing a paper with bits on machine learning (presumably written by machine learning experts), I would ask machine learning experts I know to verify the claims I was not familiar with.

I do know that a lot of ML algorithms become, at the low-level, matrix operations. Matrix operations are typically not embarrassingly parallel. Now, I can imagine a bunch of independent matrix operations, but whether you'd call the entire problem "embarrassingly parallel" would depend on how coarse the independent bits are.

In many cases you can train multiple instances of a neural net simultaneously in increments, average the weights together, and repeat. Making the underlying compute more efficient is an interesting problem.

Author here - in this case we are talking mostly about ~halving the time of a single minibatch in training a variety of standard neural networks. That type of training tends to be very hard to distribute. The servers also have 8 gpus in a single host, which is an additional performance boost and isn't included in the 2x above.

I'm curious why this claim (distributed minibatch is hard) continues to be repeated. Am I misreading the state-of-the-art? There was a paper at Interspeech this year [0] where Amazon showed excellent scaling (54X speed up when distributed across 80 nodes) without loss of accuracy.

[0] http://www.nikkostrom.com/publications/interspeech2015/strom...

This is not really true. ML applications in general do not scale linearly. There is the systems (scaling) overhead, and then there are the algorithm payoffs, which start to diminish depending on the algorithm. If they went from 4 to 8 GPUs, the only thing they can guarantee to double is how fast they burn power.

Can we please not call open sourcing a build plan of off the shelf components, open-source "hardware design". I don't think backblaze called their case designs open source hardware. It just doesn't fit when we have open source circuit board designs. How about open-source build specification instead?

I agree we should differentiate between the types of open-sourcing. Open-source hardware should either be the umbrella term or the specific one for open-source circuits. Open-source build is one phrase for this sort of thing. Maybe configuration would be another.

Seems tricky given we usually call a collection of components on PCB's and such a computer, hardware design, server, or so on. Putting open-source in front of them doesn't make it clear how much is open-source. That might be what we need to resolve. Anybody have ideas?

Yev from Backblaze here -> Call it whatever you want to call it, as long as it's not "late for dinner". Wakka wakka wakka!

Do you just watch ycombiner, reddit, and other sites all the time or do you use something to crawl the sites to see whenever someone says "Backblaze"?

Hi! No, I use a program for that, it's actually kinda cool, though I don't get "alerts" I just monitor it. It's mention.net. Highly recommend it if you have a lot of internet chatter you want to stay ahead of.

Funny story, Ycombinator is one of the reasons I got it. Since there's no great way to keep track of threads, it was a way for me to catalog and follow up with threads in these posts.

Interesting. Thanks.

I think you are completely underestimating (or misunderstanding) what this is.

The Open Compute designs (which is where this will end up) aren't just a list of parts. They include things like blueprints for custom chassis, circuit boards for power supplies and even designs for solid state storage devices[1][2][3]. The also include the source code for custom management software[4].

While it is true that they don't generally design their own logic boards, it is quite a long way from a list of parts.

I don't think backblaze called their case designs open source hardware.

You'd be wrong about that: This new Storage Pod performs four times faster, is simpler to assemble, and delivers our lowest cost per gigabyte of data storage yet. And, once again, it’s open source.[5]

I think "open source" is completely appropriate here, and to be honest I don't quite understand your objection.

[1] http://www.opencompute.org/wiki/Server/SpecsAndDesigns

[2] http://www.opencompute.org/projects/chassis/

[3] http://www.opencompute.org/projects/power-supply/

[4] http://www.opencompute.org/projects/hardware-management/

[5] https://www.backblaze.com/blog/backblaze-storage-pod-4/

I wasn't saying it shouldn't be called "open source". I was saying it shouldn't be called open source "hardware design".

Hardware design isn't limited to gates, PCB traces and ASICs. This seems remarkably like hardware design to me.

As much as you might not agree, plans for building a birdhouse constitute hardware design too.

Plans for building a birdhouse are completely incomparable to what the parent is talking about. This would be like me having an "open source" birdhouse design where the birdhouse is made of glued-together Intel™ processors.

Sure I've opened the schematics, but unlike a birdhouse made from wood you have zero chance of being able to build one from scratch on your own, or sourcing the Intel™ processors from another vendor.

This is the same reason for why if I release the binary of a program produced with a secret compiler as "open source" I'll be laughed at, it completely goes against the spirit of the thing, which is that anyone should be able to change any part of the thing I've released.

It's good that they're doing this. But it's completely incomparable to actual fully open source hardware.

Well IMO "open source hardware design" can have items in their bill-of-materials whose design themselves are not open source. I think that's where we'd end up disagreeing on terminology.

Couldn't one use the same reasoning to argue that open-sourcing an iOS app isn't really "open source", since you need proprietary hardware to properly run it?

I think the crucial difference is that when you talk about "open source X" it means that any aspect of X can be modified as you please. Obviously when you're talking about an open source iOS app it's understood that the underlying OS isn't open source.

Since there is hardware that is actually open in that sense it's worthwhile to distinguish whether you're talking about open hardware or merely open plans for assembly of closed hardware.

Open source gives you the ability to fork the design. An Open Source birdhouse made of Intel processors could be forked and made from AMD ones.

This is some kind of confluence of the values of FSF captial-F "Free" style software with the more pragmatic goals associated with open source.

We've eclipsed the usefulness of this birdhouse analogy, but suffice to say being able to replace one proprietary commercial vendor with another doesn't make something open source.

    > This is some kind of confluence of the values of
    > FSF captial-F "Free" style software with the more
    > pragmatic goals associated with open source.

Well, pragmatically, if you needed to alter the hardware of this "open source" hardware what percentage of the total electronic complexity would you be working with? I'm going to aim for a conservative one-digit percentile.

There's a real and unambiguous line to be drawn in the sand here. If you get "open source" plans for hardware that you can't replicate with third-party fabrication & your own materials it's not open source. It's just a recipe for assembling commercial proprietary components.

It's awesome that people publish those recipes, but don't call it open source hardware and dilute the meaning of hardware that is genuinely open.

Don't conflate 'electonic' with being the only valuable thing here.

Some of the most active areas of research in data center design are around the mechanical design to maximize cooling.

Why not?

Wikipedia defines Computer Hardware as:

Computer hardware (usually simply called hardware when a computing context is concerned) is the collection of physical elements that constitutes a computer system. Computer hardware is the physical parts or components of a computer, such as the monitor, mouse, keyboard, computer data storage, hard disk drive (HDD), graphic cards, sound cards, memory, motherboard, and so on, all of which are physical objects that are tangible.[1] In contrast, software is instructions that can be stored and run by hardware.

It's difficult for me to see how this doesn't fit.

The point is, this does not include the design for any of those components. It would be open source hardware design if it included the design of a monitor, mouse, keyboard, computer data storage, hard disk drive (HDD), graphic card, sound card, memory, motherboard. Open source design of any of those components would be open source hardware design. As it is, what they provided uses closed source hardware components. The hardware parts? They're nvidia cards and intel processors -- not open source. In this case it is the build specification that is open source. I guess there is a blurred line between the two. I would call circuit board designs that include a closed source chip or two as open source hardware. I just think the line should be lower level for it to be called "hardware design" as opposed to "build specification".

Open spec?

Open planning?

Open what???

Open sharing??

Open source open source open source. Design, build, specification -- anything but hardware. The hardware is the part of this that is not open source.

From Backblaze's latest storage pod blog post: "As always, the Storage Pod design is open source...."

More at https://news.ycombinator.com/item?id=10712195.

How come they aim for GPU technology rather than for neuromorphic computing ASICs?

Just a guess, but they project the cost of large scale GPU manufacturing to decline faster than ASIC due to the shared applications of gaming/VR?

You would think AI would be bigger than VR.

Edit: going with GPUs makes sense now, but it doesn't indicate a big bet on Facebook's part. I don't think opensourcing a server cabinet is that interesting, but I'm a hardware engineer.

In 10 years I think more people will touch AI than VR but I don't know how that translates to investment.

Gaming is a proven market. Capital hates risk. (Real capital, not the fringe type we talk about here.)

AI might be bigger than VR, but the timeframes are messed up. VR is an engineering problem now; I don't think there are major field-shattering breakthroughs needed to make reasonably realistic, cheap VR within the next 5 years.

AI is nowhere close to strong AI or reasonable general-purpose intelligent programs that don't require loads of human tuning. It will take several breakthroughs.

Which neuromorphic ASICs are you thinking of? GPUs are fast and they work now. Some "neuromorphic" chips, like IBM's True North, are just low-energy GPUs, so neuromorphic is in the eyes of the beholder...

This is not very accurate. The true North chip places memory next to compute elements on chip. GPUs have most memory off chip. The true North hardware is specialized to neural net stuff, and doesn't allow general floating point compute, have any sort of instruction set decoder, etc. That turns into big performance and power wins.

Contrary to what you might read in puff pieces from IBM PR, all existing "neuromorphic" hardware is terrible. The neural nets that work don't look anything like the brain, and neural nets that look like the brain don't work.

People constructing neural nets that look like the brain are engaged in cargo cult science, because the truth is we have no idea yet how the brain works and attempting to imitate it without knowing that is doomed to failure.

I'm curious to hear your opinions on Numenta's work on emulating parts of the neocortex.

He just gave it to you. Their system is not really competitive in any metric people are measuring.

A good measure of any of the techniques are the results. We see record object recognition and speech recognition results. How many of these results are from Numenta or IBM neomorphic chips? Most successes have been on deep learning architectures over GPUs (and large datasets). While these fancy architectures may have their applications, they have nothing to do with recent advancements in last 3 years.

To be fair, the goal of those new computing architectures is not to advance algorithms. That at best is a side benefit.

Those algorithmic performance improvements seen in DNN have come from improved datasets and training. Numenta and IBM arent focusing on training AFAIK. Google's Quantum Annealing [0] is the only hardware I'm aware of focused on training, although there are rumors Nervana Systems may produce something [1]. I'm sure there are others; accelerating training of DNN isnt a particularly new idea.

The goal of these other computing architectures is typically to provide lower power, higher frequency/lower latency, or smaller form factor execution of trained models, but there is a question of how much value they can provide over more conventional chips to be worth the chip design costs.

However without these architectures becoming as mainstream as say a GPU, I think we will continue to see advances come from the typical everyday computer. The ML community seems to be much more democratic than others.

[0] http://googleresearch.blogspot.com/2015/12/when-can-quantum-...

[1] http://www.nervanasys.com/about/

Numenta benchmarks their algorithms, along with others, on real-time streaming data: http://numenta.com/numenta-anomaly-benchmark/

I agree that neuromorphic is often specious, but it's not fair to say that neuromorphic hardware fails if it can do non brain like computations more efficiently than conventional hardware.

The thesis may be misguided at times but building non-traditional hardware that provide significant efficiencies is a wonderful thing. Sometimes it takes a random vector to get off a local maximum.

For example, did you know Synaptics, the guys who developed the touch input you're probably using started as a crazy neuromorphic thing from Carver Mead? [0]

While most of the IBM press is garbage, they do seem to be making progress running conventional DNNs on their hardware, showing backprop in low precision, and suggesting progress on CNNs. [1,2]

[0] https://en.wikipedia.org/wiki/Carver_Mead#Touch

[1] https://papers.nips.cc/paper/5862-backpropagation-for-energy...

[2] http://p9.hostingprod.com/@modha.org/blog/2015/12/nips_2015_...

This is off topic, but Synaptics touchpad drivers have been the worst part of using a Windows laptop for years and years. Microsoft finally had to take matters into their own hands and write touchpad drivers themselves in Windows 10 because Synaptics just couldn't do it right.

Back on topic, that paper is pretty interesting, though MNIST is a toy problem. If they could run any of the ILSVRC winners then I would be impressed, but I can't help but think that if one wants to run CNNs one would be much better off designing a CNN chip to begin with rather than a wacky spiking architecture and trying to shoehorn a CNN into it. To my knowledge nobody has yet fabricated an ASIC specifically for CNNs so there's no way to fairly compare their approach to something like that.

Both good points. Yann LeCun (et al) have developed CNN FPGA designs where they project future ASIC performance, but I'm not aware that this has ever been taped out, nor any power numbers released:

http://yann.lecun.com/exdb/publis/pdf/farabet-iscas-10.pdf

IBM has been working on these chips from 2009 I think, before DNNs really became popular again and I bet they would change a few things if they were designing from scratch today - especially better support for CNNs. The cost of chip production for something like True North is very high (5.4B transistors!) so I suspect they are trying to squeeze as much out of it as they can :)

Basically, you need hardware designers and a ton of money to develop ASIC's that will accelerate as good as current GPU's. I've seen designs along lines of SIMD's, DSP's, etc that might be a nice alternative in custom hardware. Still tons of money to get them to work. Doesn't even count developing compilers and such to utilize them properly.

Existing tools on GPU's are easier despite me wanting more work on FPGA's and ASIC's in this area. ;)

I was heavily into neuromorphic hardware ~5 years ago when the hype seemed to be at its peak. We funneled a load of time and money into it. It was working okay for what we were doing and everybody was generally happy.

Then the NVidia Tesla C2050 came out and CUDA reached a somewhat stable release. The neuromorphic plane has crashed into the proverbial mountain.

EDIT: Maybe I should elaborate - we accomplished the same task using the Tesla card, but it was about 100x faster and each card was $2500 and usable for other tasks. The general rule of thumb became: design a neuromorphic system, wait for the next Tesla chip, then simulate it in software.

Is this a machine with GPUs? You can order a similar looking design from acmemicro since last 4 years atleast?

Do you think there might be a reason that Facebook invested time and money in to producing this design, rather than buying from acmemicro?

Good work FB on another great deliverable for open HW configurations and improved datacenter design! :)

I was hoping it was the AI that was designing the hardware.

Reply on news.ycombinator.com