V100 Server On-Prem vs. AWS P3 Instance Cost Comparison

I'd imagine this is part of having a system administrator.

Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent. My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server.

> - Time (and cost) it takes to install software, drivers, etc

This is largely an initial cost of a day or two of setup when you start setting up an on-prem setup + for the first server of a totally new model. If you're buying one, then sure, you need to factor in a bit of time. If you're buying more, then if you use a competent admin the second one should be a matter of inserting boot media or (preferrably) configuring PXE booting, and picking an IP.

I agree with a lot of your points, especially the fact that TCO is practically impossible to calculate, is highly subjective, and should ideally include opportunity costs which, almost by definition vary from person to person.

I've definitely thought through some of your points. First I want to say that if your load is spiky you have no business buying hardware. No matter how you slice it, you're simply not going to be able to get hundreds of GPUs of throughput with an 8 GPU machine so it's not possible to get the same "product". So for those doing short bursts of hyperparameter search, stick with cloud.

But, like you said, for those with high utilization and known utilization patterns, it makes a lot of sense to go on-prem. Let's just say pretty much anybody who is buying a reserved instance from AWS should consider buying hardware instead.

>- Cost associated with finding an admin who understands how this thing works

Pricing based on what co location facilities will provide you without much search and is pretty generous. $10k per year per server is silly high and should cover that search cost.

> - Try before you buy

You can try essentially this exact machine on AWS. That instance and hardware are almost exactly the same. Now of course this doesn't apply to many instance types.

> - Time it takes for the box to be built, shipped and sent to the data center

5 business days is our mean lead time for that unit. Most workstations are 2 days.

> - Time (and cost) it takes to install software, drivers, etc

It comes with that stuff pre-installed as mentioned in the article. See Lambda Stack for driver / framework / CUDA woes: https://lambdalabs.com/lambda-stack-deep-learning-software

> - 3-year? What is the useful life of this? When does it seemingly become obsolete?

I'll admit it's a long time but 3 years is about right for GPUs from my experience. I personally purchased new GPUs for my workstations in 2012 (Fermi), 2015 (Maxwell), 2017 (Pascal).

> - AWS will continually upgrade their hardware and you keep paying the same

Yes, but only if you never commit to a reserved instance, in which case costs are 3x higher. If you buy a reserved instance you don't get upgraded hardware.

> - Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

Yea, most of these are people who are already using GPUs really often for internal training infra and are finding it expensive.

> - Returns, breakage, warranty in case of a hardware failure

Our hardware comes with a 3 year parts warranty. Of course, this doesn't pay for your time lost but it's pretty rare to see parts fail and the $10k / year more than covers in-colo swap outs.

I agree with all of your points. I don't think that we really disagree with much here. TCO is hard to calculate :).

Admin:

Time (and cost) it takes to install software, drivers, etc

These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker.

Maintenance / Capitalization / Finances:

3-year? What is the useful life of this? When does it seemingly become obsolete?

With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly.

AWS will continually upgrade their hardware and you keep paying the same

True, but this concern is mitigated by slow rate of GPU obsolescence.

Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)

Can you say a little more about this? Not sure what you're getting at.

Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate.

Returns, breakage, warranty in case of a hardware failure

Any reasonable hardware provider will include an option for 3-year warranty.

I think they used the 3-year reserved instance pricing, meaning you can’t just turn it off after a year and stop the cash flow out (at least not at that price).

Many benefits of running infrastructure in the cloud are lost for offline batch processing jobs. Training machine learning models doesn't require low response times, high availability, geographic proximity to clients, etc. Yet, with cloud, you're paying for all this extra infrastructure.

The main benefits of cloud for tasks with machine learning type workloads are cost saving (if utilization is low) and no electrical set-up.

On the other hand, cloud is extremely expensive for groups that require high base levels of GPU compute. The article is arguing that such groups can save a huge amount of money by moving infrastructure on-prem.

Or like saying that if you spend 20 hours every day in an Uber buying a car would be cheaper over 3 year term.

Nobody is disputing that buying similarly specced hardware on the market is likely to have a cheaper TCO, the point of cloud providers is that also have a bunch of stuff attached to the servers, which turns to be pretty important.

If you're buying more than one, that'll easily cover enough to cable up whatever network you want between them.

Many colo providers also have their own cloud services, and nothing stops you from using S3, SQS and the like, though of course if you use AWS services "from the outside" you do need to factor in their extortionate outbound bandwidth charges.

As bubblethink already mentioned, the article is arguing that batch processing jobs (e.g. those characteristic of Deep Learning) don't benefit substantially from cloud infrastructure.

High availability, edge processing, fast network upload, and the lego blocks for creating redundancy are moot.

Most training algorithms automatically recover if a node goes down. That covers all required failure handling.

Most DNN model training workloads are lumpy and transient.

"Everyone" seems to think they have massively spiky loads, and almost nobody does. There may often be demands that could spike (e.g. because "someone" has not randomized the start times for batch maintenance processes through the night but start them all at the same time), but more often than not the spikiest demand comes from batch workloads where nobody cares if it takes a bit longer. In practice when constrained that often simply smooths out the overall capacity use.

(of course that does not mean that there aren't people with spiky enough workloads for it to make sense to put at least some of their workloads on VMs for cost reasons; but measure first)

I remember, though, that NVIDIA disallows in EULA the use of nvidia driver blob in consumer gpu in datacenter environment. Time will show how legally enforcible is this restriction.

Most of this is due to arcane ordering processes both on client and supplier side.

The price difference can be substantial enough that you might simply be better off having it "over provisioned" off hours.

Also, not having "reactive capacity" might simply mean that your 95th percentile goes from 100ms response time to 300ms. Which again, might be a more cost effective approach.

I think the initial development and ongoing cost of maintaining that reactive capacity is also substantial enough to be considered.

Most of the apps that truly need the elasticity should probably going hybrid anyways. Baseline on dedicated hardware, spikes on spot instances.

Almost everyone thinks they can; very few small businesses have the sysadmin/devops to pull it off.

Every 6 months or so a hard drive fails (out of 16 per server), no other components have failed. 10 machines are 6 years old (test system as it's out of warranty), 10 3 years old, 10 new.

There's also 20 or so other servers under different loads, I've not had anything fail other than hard drives.

When I started the job, there were spare power supplies for some 10+ year old servers in storage, so I'm either lucky or reliability is improving.

I used to work for a VFX company where we would try and get as close to 100% utilization out of the farm.

We had machines that were 4 years old, still merrily plodding away. (not too many mind, they ate more power than its worth.) The things that tended to fail were Harddrives and fans.

AWS is great for convenience when you can afford it, but it is a really expensive solution, even when factoring in the extra things you have to deal with to rent, lease or buy dedicated servers.

You'll need additional off-site backup, but that's starting to get out of the scope of the article.

Anyway, despite my belly-aching, it was an interesting read.