Build a fast deep learning machine for under $1K | Hacker News Reader

Build a fast deep learning machine for under $1K

oreilly.com

147 pointskqr261 comments

Contrary to the claims there, the CPU does matter: if the network is relatively small, it's easy to get bottle-necked feeding the GPU.

Same thing applies to storage, for the same reasons. There's a reason AMD is selling deep learning cards with Flash drives built in.

The choice of CPU cooler is silly. The CPU chosen is a 65W model, which will be cooled quietly even by the boxed cooler (and we don't need to care about overclocking with a non-K). By his own admission, the rig draws 250W, so the idea that 650W must be cooled is nonsense, and even then, only a part of that is due to the CPU.

It's probably worth looking more closely at the GPU, notably, if that cooler stays quiet when the card is under full load.

You do not need a monitor, keyboard or mouse to use such a machine. You can just ssh into it. You might need to borrow a set to set up the BIOS.

If you "run out of memory for your applications" on the GPU, have you considered simply lowering the mini-batch size? (See, this is why we got a good CPU, it removes the hurt from feeding smaller batches)

There's a reason AMD is selling deep learning cards with Flash drives built in.

For those not familiar with them, this is the Radeon Pro SSG[1]

This has a built in M2 drive, but this is used as video card memory, NOT main storage. It could possibly allow big (HUGE) models or batches, but doesn't prove fast IO is often a bottleneck.

Practically no one[2] is using these for deep learning. They are marketed to the oil and gas modelling and visualization market and notably the product page (linked above) makes no mention of neural networks or deep learning, and it has never been mentioned in AMD's publicity.

Yes, it might be good for deep learning. And yet, modern deep models do want more memory. But there is a lot that needs to happen for this to be useful: AMD needs to release something like CuDNN and they need to make sure OpenCL is supported as well as CUDA is.

[1] https://arstechnica.com/gadgets/2016/07/amd-radeon-pro-ssg-g..., http://www.amd.com/en-us/press-releases/Pages/amd-radeon-pro...

[2] I'm sure you'll find someone. But show me a published paper or any kind, or even benchmarks showing the use of the extra memory somewhere.

doesn't prove fast IO is often a bottleneck.

It's not often a bottleneck. In fact it usually isn't. But when it is, you're left wondering why you didn't get an SSD. They're cheap enough now.

MIOpen is AMD's cuDNN alternative.

Which does not actually exist. It's announced but unreleased.

My home server has an i7 4790K, 32GB DDR3 and... a GTX 1080 Gamer Edition that's doing... well, bugger all. Why is the graphics card in there you might ask? Well, it was sitting on my desk collecting dust as I wait patiently for a Pascal driver for eGPUs on macOS but Nvidia had delayed these for long enough that I thought I'd chuck it in my home server, it is being used by ffmpeg for x265/HVEC transcoding with Plex media server but I'm really not even touching it's capabilities - not even close when transcoding a 120GB+ 4K 120FPS video and streaming it wirelessly to my TV that's running a Plex client. The server is running CentOS 7 and has 10x 8TB SATA drives in BTRFS RAID 10 for storage, two Intel NVMe 750 series 1.2 TB SSDs for the OS as well as a couple of 2TB Crucial MX300 SATA SSDs that aren't currently in use for much.

So, I have been sitting here wondering how I can make better use of it while it's practically parked in my server while I wait for some Pascal drivers to drop. I'm not so interested in robots / live video processing but it'd be neat if I could leverage its power for something fun or to assist the servers compute performance in some other ways that might be useful or interesting. I was running KVM with PCIe pass through passing the GPU to a guest VM running Steam as a steam streaming box but I got sick of having an OS on the network that I had to worry about Virus's, Malware and annoyingly slow and unreliable updates being installed when you least want it to happen and I got lazy and went back to playing PlayStation when I needed some time out from constructive work / research.

Open to ideas to any software I could tinker with if anyone has any interesting suggestions?

Not something for you to work on, but something you can do in the mean time:

https://en.m.wikipedia.org/wiki/List_of_distributed_computin...

Can you ballpark today's price tag for such a setup? There seem to be several pricing tiers floating around in this discussion and I'd appreciate the chance to see how your setup fits in without having to shop for it.

Well it's all cheap kit with the exception of the disks, I guess you could just go online and have a look at what those parts cost you in your country etc...

I think there should be a disclaimer here that while, yes, this will work, it will quickly become suboptimal (perhaps even dysfunctional) for a lot of work as you scale your hobby into something resembling more professional work. I personally would not start with a thousand dollar machine if you have any intention of doing serious computation in the future. While you can trade out parts pretty easy on a custom built machine, you don't want to be in a position where you have to trade out most of the core parts 6 months in.

For reference, I built a home PC that I successfully do deep learning and data analysis on (mostly tensorflow and scipy stack) for about ~$10k. It's liquid cooled, has 15 fans, four radiators, an i7-6900K CPU, 128GB RAM, four GTX 1080 GPUs (controversial), four TBs of HDD space and 1TB of SSD space. I don't recommend you start with this at all, but my point is that porting your hardware from point A to point B will be a pain if it comes to it.

I used the guide here as a reference about 8 months ago when I built it: http://graphific.github.io/posts/building-a-deep-learning-dr.... My purpose in doing this was, essentially, to pay for electricity rather than AWS/GCP/Azure compute resources (and in that regard it's been very successful!).

I know I'm hijacking a thread here to talk about building home machines for professional deep learning work when this story is clearly not intended for that, but I wanted to throw in this perspective so that it's understood this is very different from just "build this machine to start out and upgrade it later." There's a law of diminishing returns here, but in general my point is that I do not think this is a minimum for "start doing deep learning effectively at home." If you want to learn hands on deep learning cheaply, my opinion is that it would be more efficient to use compute resources from a cloud provider before diving into this with a home-based custom machine.

tl;dr: The demographic of folks who probably want/should/need to build a home deep learning machine probably has little overlap with the demographic of folks who want to do it non-professionally, or at least with only $1k in resources.

so do you earn money out of that 10K investment on machines? I didn't read the article but what's the benchmark on deep learning machines? Still TFLOPS?

Yes, and yes.

How do you make money off a deep learning rig? As a consulting data scientist?

Serious question, why build a 10K$ machine like this rather than just spinning up some AWS instances? -edit- I know you mention wanting to pay for electricity rather than AWS. But that doesn't necessarily make financial sense.

Many prioritize protecting their data, eliminating the possibility of using the cloud. This is usually a business use case though rather than personal.

This is correct. I handle my own (daily) backups as a result, which are end-to-end encrypted from the machine.

If it's a business use case severe enough that you can't trust AWS, you aren't going to wast time scraping together a cheap $1K or even $10K build while your $1K-$10K/day team is idle. You'll go looking for accessible powerful systems, and be willing to replace hardware as needed.

This rationale is exactly what the AWS team has seeded in the community. I think it's time we quit parroting an inevitable conclusion that all things will and must go to the cloud.

On the other hand, by the time anyone starting just now learn enough ML to be dangerous, the price of the $10,000 hardware would drop to $5,000 or less, so starting with the cheapest yet decent option seems a better approach...

Fair point. I'm just quibbling about where "decent" begins, and adding in the point about ease of upgrades. Replacing a GPU is pretty easy. Replacing a motherboard because you don't have enough PCIe slots will require tearing down the majority of the machine and rebuilding it.

The latter is more likely to happen if you start down this path at rock bottom prices.

I began building one a year and a half ago, starting with a 5930k, 1x980ti, 32GB. This put me in a position to incrementally upgrade it starting at ~$2.5k. That being said, at this point the smallest AWS p2 instances are equivalent to my current setup, and it would take 150 days worth of running my hardware to make it cost-effective vs. AWS.

When you build your first DL box you also get a moral boost. If you think every second is counted and penalized against you, you might use it less (such as when you're using cloud servers). But if it is already yours bought and paid for, then you can afford to run any crazy idea. A good compromise would be a local DL box and extra virtual machines on demand, for when you need to run extensive hyperparameter searches. If you make the leap and invest, at least you commit your money into the direction of ML, strengthening your resolve.

On the flip side, dropping $2.5k for a hobby you haven't investigated yet and then looking back 6 months later on the computer collecting dust will likely dissuade you from investing any time into potential future hobbies. I play piano, and I'd never encourage anyone to go drop $2.5k on an electric keyboard just to provide a motivation. A decision to pick up a new hobby/skill should be based on desire, not a handcuff.

Are multi-GPU systems worth it until you have a TON of experience building parallel models? Almost all of the reference architectures I've seen are essentially serial, and TF doesn't have good data-parallelism built in to make use of N GPUs, N > 1.

Maybe this is a stupid idea, but if you had two GPUs, doesn't that basically mean, in terms of good parallelism, that at least you can train two models/networks at the same time, roughly?

Because as long as you're in the research and development phase, that'll help cut your coding/training/testing/adjusting cycle. I assume you'll be tuning your hyperparameters, perhaps on somewhat smaller test models, but they will still take half a day or so to train? That means you can try (almost) twice as many hyperparameter configurations in the same time. It still helps to spin up a second test with a different selection of parameters, even if you haven't gotten the results back from the first test, right?

I wouldn't recommend 4 GPUs unless you are very confident the algorithms you are going to run will parallelize well, or that you need to run extensive parallel trials for various hyperparameters. It takes special effort to make use of multiple GPUs.

Can you share your PCPartPicker list? I'm curious how the costs breakdown.

So what you're saying is that if you start with the thing in this article, and then want to upgrade to the thing you described, don't expect it to cost $9k, it'll cost $10k :-)

(and as a bonus leftover: a nice machine to donate your local volunteer hackerspace, youth tech center, school etc etc)

I don't see the problem, as long as it's roughly 9-10x faster for an embarrassingly-parallel task of larger size, that 10% isn't going to make a big dent, is it? :-)

If I were building a GPU box on the cheap, I'd start with a used Dell Precision T7xxx series off of Ebay for a <$300 including RAM and a Xeon or two. They have 1100 watt power supplies and lots of slots and the RAM is usually ECC. Odds are it will come with DVD burner and possibly a hard disk and a Windows license...again all for a couple of hundred dollars.

Leaving $700 for GPU's while providing reliable high end hardware pretty much designed to run GPU's for the base platform.

I'm not sure why mITX was chosen - in my experience it's almost always more expensive and leaves no room for additional GPU expansion (Or network adapters/whatever card you might need) in the future. You can get 1151 mATX motherboards for more like $60-70 on Amazon instead of the $125 they paid.

Also, paying $125 for 16GB of ram but not spending the extra $20 to have your CPU be able to overclock? I'm not sure where OP is building but I can find that much RAM for $20 less than they paid, and overclocking isn't really that difficult or unreliable these days.

I was wondering that too. For something like this maximum room for future expansion seems good. In the past I've used mITX to build low-power, low-noise machines, rather than building a high-powered processing beast.

On the other hand, it made the article more interesting for me. I can't really justify a home machine learning rig, but I was thinking I might soon replace that old mITX machine.

(edit - just reached the pics - really not sure why you'd go with a mini form factor and then add a 6" heatsink/fan!)

I also prefer mITX and I have one I've tried to optimize for low noise and efficiency. Just seems ill-suited for this purpose.

Most people don't want a giant box under their desk. If you're building a machine learning rig for under $1,000, you don't need room for expansion because additional $200, $400 or even $700 cards are a useless expense at that price range.

You don't need a powerful cpu for gpu loads.

In Europe it's also very common to just choose the components in a web shop and have them assembled, also for about $1000. (Not looking at Dell)

Anyone interested in deep learning should go that path instead of burning their money on Amazon.

+1 on that. There are a number of online build-to-order shops here in the UK which offer a truly vast range of component combinations. Better still, the order process automatically checks for compatibility with, for example, your choice of motherboard.

Once you have the configuration you want, you just pay up and it arrives professionally assembled ready to rock. I don't think I'll ever by an off-the-shelf machine again.

I found the premium on these was quite high, although even just buying all the components from a single place added a fair amount to my recent build.

Compatibility, price checking and searching are all available with PcPartPicker (UK site: https://uk.pcpartpicker.com/) which saved me an astonishing amount of time.

The hour or two building the machine were worth the few hundred it saved me, but everyone has different priorities.

I've found that the premium depends on the store and model. Some seem to do relative pricing, others absolute. In the latter case, if you want a big rig it's best to let someone build it for you.

Often times in those cases you also get great service because a) if you build PCs for a living you're a computer geek and it's fun to build an insane PC and b) they often use their biggest systems as advertisement. At least, that's what I've seen.

+cable management is like black magic to me. If I were to take the money saved as payment for me to get it as nice as those places get it, I'd be below minimum wage.

Can you suggest some of those sites that ship across European Union? I've been searching for something like that.

I've been considering a "thing" that allows monetization of an AI-enabled machine by others who need the capacity to do larger scale testing. Anyone interested in this? My contact info is in my profile.

You won't be able to match price/expenses (fixed with hardware or variable with upkeep) of AWS/GCP due to their economy of scale. Even startups who are doing similar things sublicense from AWS.

How is it better than AWS?

But how do you scale cheaply to more GPUs?

Get a ATX instead of ITX mobo. It's likely is cheaper as well.

If by "super fast deep learning machine" you actually mean, mid tier gaming computer. Uh, sure. Don't spend more on an ITX board, when you can fit 4x to 7x the video cards in a full size ATX computer using pci-e risers, while paying less for the board.

Don't lose 5 percent of your cpu overclock or more by having an itx friendly cooler instead of a D15 noctua or better.

Choose your video card wisely, AMD does some things better than NVIDIA depending on what software you're running.

If you chose an ITX platform for deep learning fun, uh, you should really add the couple shoeboxes extra space to your platform and have 4x-7x the power available to upgrade into.

Try to bring good in the conversation.

What piece you would have use? Why? What tradeoffs you have considered?

You sound expert in the field, try to share your knowledge with the community in a constructive way so that we can all benefit from it.

I develop software for living and still I haven't any clue of what you said, while I followed quite well the article and I thought he was making reasonable choice.

Please show me better.

I'd like to second this request and note it took me years to figure out that people saying "focus on the positive" in this context would be satisfied by me phrasing things in terms of what to do rather than what not to do. Concrete examples are key for this approach.

Not parent, but I've built my own computers for almost 20 years.

As the articles author himself questions, he should have gotten the marginally more expensive CPU, and definitely the GPU with more RAM.

* The overclockable CPU doesn't just mean that you very easily could get a 10%+ performance boost without much work, but also that you often (depending on your specific chip) can lower the voltage and make it run much cooler/quieter, which is something I increasingly care about if I'm using it a lot.

* He writes that fitting the model in RAM is basically the most important part, but then saves less than a dinner out by basically halving the RAM on the GPU he bought. The chip is otherwise the same though, so performance is only dependent on whether you fill that RAM or not.

* The Noctua D15 the parent mentioned is a CPU cooler where Noctua is a long standing high performant brand, and D15 is a specific model with a 15 inch fan and comparable sized heat sink. There are of course other brands, but I myself also usually end up with Noctuas. The reason it's important is that however fast you can dissipate heat from the cpu/case, the less chance of throttling, and the larger headroom for potential overclocks you get.

Airflow and room for larger heatsinks is also why he recommended not going for an ITX. A linked benefit is again the potential for a quieter system.

I haven't gone much into ML (yet), but I currently have a system with:

* I7 6700K (the difference to I5 6600K being higher base clock and hyperthreading, which is more important to computational work than to gaming, so if you have the money, definitely go for the I7)

* 32GB DDR4 (as author mentioned, RAM is cheap). The clock/timings on RAM isn't really as important, but try to find the best you can find for a given price point.

* An Nvidia GTX 1080: It's not Titan X or Z, but almost, at less price. It definitely blows the budget for a $1000 system, but I agree that the entire 10 series is good.

If the limit is a firm $1000, I would get something like this:

https://pcpartpicker.com/list/XHV9Fd

And if more funds is available, I'd get more storage and RAM, then a better CPU, then a better GPU, then maybe bump the chassis up to an R5 (same brand), possibly another motherboard. In that order. There's always something better, so you compromise based on budget.

I've got a near identical build, and my process was the same as you. I added a decent pcie nvme drive, and spent more than I guess I could have done on a motherboard that supports up to 128G ram so I can extend it later. I'm waiting for ram to drop in price a bit, or to hit a problem where I need more than 32 sooner, before bumping that up and probably adding more storage.

It's been great sshing in from my laptop, submitting a job that completes far more quickly and keeping my laptop cool.

I kept my old 512GB Samsung 850 Pro, but will definitely get an nvme when I need more space/speed. Getting another 32 RAM will hold me over until I'll build a new one, so that part I'm not worried about.

Yeah, it's supremely fast, my only regret is not having enough time to do something fun with it. For work I'm stuck with the clients platform approved machine, which is not what I would've picked. Tough to complain, but if I'm ever between contracts, I'll likely get into some fun project.

I have a machine that I am using for deep learning and it has 32GB of ECC RAM paired with a Xeon CPU w/ Quadro, and that 32GB of RAM is gone almost instantly. Now I'm looking at building another machine as it has gotten to the point where I am spending a great deal of time waiting.

I would be interested to see what other people are using for their setups, and how that can differ for things like high-resolution style transfer or generic neural networks, etc...and at what point they have to switch from geforce/quadro cards to tesla.

Was years ago when I build my last PC and I'm out of the game.

I'm tempted in build a hackintosh:

https://www.tonymacx86.com/threads/hackintosh-cutting-edge-k...

- Intel i7 Kaby Lake - No decided on motherboard. The one that cause me less trouble (for hackintosh) is fine. - GTI 750ti (have) or buy a pascal nvidia. - NVMe drive if possible - 32 GB RAM. - Probably a Thermaltake CORE P3 case. Not decided.

I was thinking in use a Liquid Cooler but wonder if the Noctua could be better/less noise?

> I was thinking in use a Liquid Cooler but wonder if the Noctua could be better/less noise?

Depends, is your option of water cooling an All-In-One-solution that have become popular in recent years? Their performance is on par or slightly better than a large heat sink + large low speed fan(s), but they're not generally quieter, as you still have fans for them, as well as a pump.

I considered those options when building mine too, and as I wans't too enthusiastic about assembling my own water cooling system, I went for a large air cooled heatsink instead (the D15). No risk of leakage or pump failure, and proven performance/low noise.

I was planning in use Corsair Hydro H60 (http://amzn.to/2k9a9O1).

However, I wish to have a quiet system, and my brother have it and it sound louder than I wish. I don't plan on overcloking.

Yeah, that's one of those AIO water coolers I was referring to.

http://www.anandtech.com/show/5054/corsair-hydro-series-h60-...

The "Silver Arrow" is an air cooler from Thermalright that is pretty equivalent to a Noctua or a Phantek or be Quiet! etc.

In that review it beats the H60 in both temperature and noise. The H60 is more than twice as loud.

So yes, unless you assemble your own water cooling system, I'd say definitely go for a regular heatsink+fan.

Just FYI, 10 series cards aren't supported by hackintosh currently (and for the foreseeable future).

Yep, the parts I wish to use need some hacking. Supposedly kaby lake + nvme can be made to work. I don't need 10 cards, I can keep the 750ti

I run one with a 5930k, 980ti, nvme. It had huge issues with el capitan, but sierra has been a breeze. The pascal part is an issue, but running a small card/integrated graphics for the screen is perfectly fine if you're dedicating the GPU to computing.

Agree with a system you've linked. I believe 1060 has best price-to-value for deep learning in the market.

> Choose your video card wisely, AMD does some things better than NVIDIA depending on what software you're running.

Deep learning toolkits basically have two modes of operation: the CPU way, and the proprietary NVIDIA way. There is no point to putting an AMD chip in a machine you hope to use for deep learning.

Some toolkits may go through the motions of supporting AMD via OpenCL, but that's not going to be the case that they make sure works well, or works at all.

This is a problem, as NVIDIA is awful at maintaining drivers, results are being published based on "well, my NVIDIA black box decided to do this", and it contributes to deep learning veering toward a local maximum. But specifically choosing to do deep learning with an AMD chip is a pointless sacrifice.

If you're on a budget it makes little sense if any to buy things like a motherboard, CPU or memory new.

Reply on news.ycombinator.com