New P2 Instance Type for Amazon EC2 – Up to 16 GPUs | Hacker News Reader

New P2 Instance Type for Amazon EC2 – Up to 16 GPUs

aws.amazon.com

223 pointsjeffbarr123 comments

$0.9 per K80 GPU per hour, while expensive, opens up so many opportunities - especially when you can get a properly connected machine.

Just as an example of the change this entails for deep learning, the recent "Exploring the Limits of Language Modeling"[1] paper from Google used 32 K40 GPUs. While the K40 / K80 are not the most recent generation of GPU, they're still powerful beasts, and finding a large number of them set up well is a challenge for most.

In only 2 hours, their model beat previous state of the art results. Their new best result was achieved after three weeks of compute.

With two assumptions, that a K80 is approximately 2 x K40 and that you could run the model with similar efficiency, that means you can beat previous state of the art for ~$28.8 and could replicate that paper's state of the art for ~$7257.6 - all using a single P2 instance.

While the latter number is extreme, the former isn't. It's expensive but still opens up so many opportunities. Everything from hobbyists competing on Kaggle competitions to that tiny division inside a big company that would never be able to provision GPU access otherwise - and of course the startup inbetween.

* I'm not even going to try to compare the old Amazon GPU instances to the new one as they're not even in the same ballpark. They have far less memory and don't support many of the features required for efficient use of modern deep learning frameworks.

[1]: https://arxiv.org/abs/1602.02410

Still seems much better to buy your own gtx 1080 for $700 which you would have spent in a month playing with parameters on these instances.

And at the end of the month you still have a modern GPU to play video games on etc.

Of course if you have money to burn, these are great.

I question the amount of people that would be buying these for research AND gaming.

I saw quite a few asking for PC builds for both. Might not be too common, but is also not something that happens only once a year.

The demographic overlap between "gamer" and "machine learning researcher" is likely very high.

Likely how??

I guess: University students.

University students can afford $700 graphic cards?

I would assume they have (or have access to) a computer fitted out in a research lab or something which would be seperate from their "gaming rig".

For many students, I suspect the "computer fitted out in a research lab" is some dell desktop box someone shoved a gaming GPU in 3 years ago and that they have to share with 5-20 other students. Better not break it.

If you are into the topic and want to work on things outside of assigned projects or course work, upgrading a personal machine doesn't seem far-fetched if you can make room in the budget for it. $700 isn't cheap, but doable for many as a one-off purchase if they want it.

> Likely how??

Those who are predisposed to buying & installing high-end graphics cards are likely to be either gamers, interested in machine learning or both

That's not quite correct - it might be fair to say 'machine learning researchers are likely to be gamers', but it's not fair to say 'gamers are likely to be machine learning researchers'

or P(X|Y) != P(Y|X)

Yes, that would have been a better way to state it.

Im assuming most machine learning researchers who have big computers with expensive graphics cards for research wouldn't have it in a personal computer they can easily game on.

While I'm sure your statement is true for persons, I doubt its true for the computers themselves.

This sort of baseless statement happens all the time in our field. Gaming is not technical. Gaming has no relationship with programming, unless you are programming games, which is a completely different subset of people. The misuse of "nerdy" to refer to people who consume a mainstream source of entertainment is baffling to me. Nobody would say someone is nerdy for using social media for hours a day, and yet they both require the same amount of technical proficiency.

Most people I know who got into the field via gaming were either completely incompetent and have since left, or do not label themselves as 'gamers' anymore.

The common label is "graphics programmers", a.k.a. "demoscene people." There's a large overlap between those and a certain kind of gamer—the ones that create Tool-Assisted Speedruns, for example.

It's a simple matter of demographics.

Who are the majority of machine learning researchers? PhD students in CS (aka majority male) between 22 and, what, 26 years?

What percentage of male computer science students between 22 and 26 years DOESN'T play games?

By the way, I own a 1080 since two weeks and I can't overstate how powerful this thing is. Even if you are not into gaming, getting a 1080 is still a considerable option if you want to experiment with deep learning.

Get an GTX 1080 for your local box and use it to write and test the model, then run in cloud for training. Both are necessary, in fact, they are complementary. If you want to run random hyperparameter search (train 50 times) then you benefit from many GPUs, but a single 1080 doesn't scale.

You can set up the volume a CPU only machine (even on a free instance), and then just launch that volume with a machine with these big expensive GPU's on it.

I did it a few months ago with a TensorFlow model I was training. Configured it for free then all the training time was fully used after configuration.

I assume it'll be the same with these new instances. I have a backlog of stuff I wanted to train but had no GPU to train it on (I travel a lot so couldn't buy a desktop, and was almost biting the bullet to buy one of those monster laptops with dual 1080's, although really I was after at least 12GB of VRAM so this is fantastic news for me)

Have you considered buying/building a desktop that acts as a server and then using ssh to connect to it from your laptop?

That's what I do.

I'm not the person you posed the question to, but what's the secure, convenient way to do that? If I want to ssh into my desktop from within my house? If I want to ssh into it from elsewhere?

Am I missing the question? Maybe, maybe not. The simple answer is build your new "server" and install some version linux, like Ubuntu. Set up the accounts and make it so you can login using a private key. Turn off remote login with password so that you can only login with the private key. Install Fail2Ban to block automated hacking attempts---almost the second you expose your server to remote traffic you will start seeing attempts. Configure your internet router so that incoming port 22 traffic is routed to the server (or all ports if you don't care about security). Signup for a free dynamic DNS service for when your IP changes. Now you can login to your home server from anywhere using a your hostname which is tied to your home server external ip address. Its secure because there is no password to brute force and Fail2Ban is killing external attempts. You can secure it further using more research, but this is the basic answer.

Nope, you aren't missing the question. That's just a lot of stuff I didn't know to do (or how to do). Thanks!

You can use ZeroTier if you need to traverse a NAT or don't have a static IP and you don't want to deal with DynDNS. That's what I do to access my work computer. It's free and works perfectly.

Yes, a lot. but I don't really have a permanent location to store the server unfortunately.

If you're a regulated financial institution, say performing complex risk valuations, you need the ECC ram. It's a consumer device vs an enterprise, professional device.

That's not the use-case for this product. You can fiddle around with a single card till you get it right and then run the real job on EC2 where you most likely order dozens of these machines and run them for weeks/months. It's rarely economical to purchase hardware at that scale, unless if you are AWS renting out computation to others. At that scale you have to think about way more cost factors than raw hardware cost.

While you're right that buying your own GPU makes a lot of sense for personal use or small tasks, if you factor in the cost of keeping a ML team waiting for the network to finish training, then it might be cheaper to invest $7000 and have it run in a few hours instead of weeks.

Also, consider the efficiency of owning such a 16 GPU beast: if you don't have tasks to run 100% of the time, then you're losing a lot by just keeping it idle. Cloud providers can reuse the same GPU for other clients in the meantime.

You will need time to buy that gpu, upgrade your PC, realise that it doesn't fit into your existing case and that you need a better power supply. Then some time to find a better place to be than sitting next to that really loud and hot PC. And it all takes optimistically days, while you can start computing on EC2 within in an hour. Plus it doesn't scale, so you cannot just speed up the results by paying some more. You may need to wait a month (and then realise the param was wrong and you need to redo it)

Surely there are some cases when you are better off computing on your own GPU, but I think in all of them you assign pretty low value to the time spent.

Azure N Series are about 35% cheaper[1], although only up to 4 Tesla k80 GPU per instance.

[1] https://buildazure.com/2016/08/09/azure-n-series-vms-and-nvi...

I've been waiting for access for a long time. I have no idea when they will accept users other than a few hand picked beta testers.

Using spot pricing you tend to get it far cheaper than that. I was using the previous GPU instances a few months ago which are meant to be $0.65 an hour. But using spot pricing, setting the maximum to about $50 so I wouldn't get shutdown in the middle of training, I seemed to spend an average of about $0.20 an hour.

Now I'm using a P2 instance to see how much faster it runs the same neural net. I set the max price to $0.50 in a spot price bid and got it immediately. I'm not sure how to find out exactly how much it is being charged however.

https://aws.amazon.com/ec2/spot/pricing/ has the current p2 spot prices

And as of 9am PDT you can get a single GPU Linux instance for less than ten cents an hour. I doubt it will last, but if it did you could get a year's worth of compute for $800 and still have the flexibility and reliability of AWS infrastructure.

MeteoSwiss uses a 12 node cluster with 16 GPUs each to compute its weather forecasts (Piz Escha/Kesch). The machine is in operation since more than a year. We were able to disable the old machine (Piz Albis/Lema) last week.

The 1.1km forecast runs on 144 GPUs, the 2.2km probabilistic ensemble forecast is computed on 168 GPUs (8 GPUs or 1/2 node per ensemble member). The 7km EU forecast is run on 8 GPUs as well.

What kinds of numerical algorithms are used for this? e.g. Monte Carlo or Finite Element Analysis?

It's a simple finite differences discretization on a rectangular grid. The computations are horizontally explicit and vertically implicit (data dependencies in the vertical direction).

Here's the scientific description of the model: http://www2.cosmo-model.org/content/model/documentation/core...

For anyone who interested in ML/DL on cloud.

Google Cloud Platform just released Cloud ML beta with different pricing model, see https://cloud.google.com/ml/

Cloud ML costing $0.49/hour to $36.75/hour, compared to AWS $0.900/hour to $14.400/hour

The huge different of $36.75/hour (Google) compared to $14.400/hour (AWS) make me wonder what Cloud ML are using, they mentioned GPU (TPU?) but not exact GPU model.

For some definition of "release". This is something they've had in beta for a while, and its only accessible to people who are already in the "beta" program for cloud ML. Unlike AWS which offers it _now_.

I believe previously is "alpha" release that required signup invitation form. Now "beta" release is open to anyone with Google Account.

Do you have a link you can point us to to validate that?

https://cloud.google.com/blog/big-data/2016/09/google-cloud-...

There are two issues, Google offers a black box service where your Tensor Flow model gets optimized. While AWS offers a generic instance which is far more flexible and can be used with multiple frameworks. Second AWS offers spot pricing on GPU instances (though not on P2 instances yet) which can offer significant discount depending on the type of workload.

I would say that while I still have to evaluate Google Cloud ML, AWS offers a far far better experience and cost effectiveness.

They already offer spot p2 instances, although I think the main spot pricing history widget isn't up to date yet. The current minimum spot price seems to be 0.09 USD/hour for a p2.2xlarge, and there have already been steep price fluctuations for p2.16xlarges in the past hour.

Oh wow they do, 0.09$ per hour thats cool! Hopefully it will take another week for others to transfer their models from g2 instances, should reduce price pressure!

I tried using google cloud (switching from AWS) a few months ago and it was the worst user experience I've ever had trying to use any software / dev environment. The UI was far worse than windows 98, and horribly confusing. And I got the impression you can only run standard tensorflow libraries, without any modification. I needed to run a modified version of tensorflow so it was a massive waste of time.

I couldn't believe how much better AWS is for running TensorFlow models than Google Cloud.

Are you talking about the Cloud ML service that was in private Alpha? Would love to hear if your opinion has changed now that it's Beta. Alpha products tend to have rough edges all around.

(work at GCP)

Speaking of... where are GCP GPU instances, without Cloud ML?

Is it possible to run customized (ie, recompiled, with new C code embedded) TensorFlow instances in it now? without that there's no point in looking.

Thanks for replying.

  All of the instances are powered by an AWS-Specific version of Intel’s Broadwell processor, running at 2.7 GHz.

Does anyone have any more information about this? Are the chips fabricated separately or is it microcode differences?

http://m.youtube.com/watch?v=JIQETrFC_SQ

I think this is the right talk by him, but there's one where he implies some differences. For instance, that because they can promise exact temperature ranges they can clock them differently. My guess is same fab, just binned differently, or maybe different packaging.

Perhaps with overclocking unlocked via microcode.

Keep in mind the Nvidia K80 is a 2 years old Kepler GPU. Nvidia launched 2 newer microarchitectures since then: Maxwell, Pascal. I would expect to see some P100 Pascal GPU "soon" on AWS. Maybe 6 months? (Maxwell's severely handicapped double precision performance reduces its utility for many workloads.)

This is a big issue because multiple DCNN optimizations in CuDNN are only applicable to the newer generations. You could be looking at an order of magnitude worse performance than expected.

> Nvidia K80 is a 2 years old Kepler GPU ... I would expect to see some P100 Pascal GPU "soon" on AWS. Maybe 6 months?

Not likely.

The AWS CG1 instance type was introduced in 2010 (NVIDIA Tesla C2050 GPU). The G2 instance type was introduced in 2013 (NVIDIA GRID K520 GPU). Now the P2 instance type has been introduced in 2016 (NVIDIA Tesla K80 GPU). At this rate, we probably won't see another GPU fleet in AWS until 2019.

I never realized Amazon was so consistently late in deploying current-gen GPUs. How lame. Sounds like an opportunity for a competitor to beat them.

Maybe they're waiting for the old ones to recoup their investment or the new ones to get cheaper?

It's not just Amazon. No large cloud provider offers recent GPUs. I believe the blame probably lies with NVIDIA.

Pretty classless how Jeff describes TensorFlow as an "Open Source library," without atributing it to Google.

Does every mention of an Open Source library or project morally require attribution to the original authors or something?

No, lol. The license stipulates the conditions, and the common licenses only have the condition that the code or final product make the attribution somewhere in the GUI.

> Does every mention of an Open Source library or project morally require attribution to the original authors or something?

No. But in my view had this been written by an independent journalist, they would describe it as "TensorFlow by Google". The project is virtually completely driven by Google, so "By Google" is a more distinguishing classification than "Open Source". We can speculate but strong impression is that Amazon doesn't want to give kudus to Google.

Not sure that Google would agree. Looking to Google's own words about open sourcing the project:

"We hope to build an active open source community that drives the future of this library, both by providing feedback and by actively contributing to the source code."

One thing I discovered recently is that for GPU machines your initial limit on AWS is 0 (meaning you have to ask support before you start one for yourself)

(This might be an issue of my account though - having had only small bills so far)

Wonder if it has anything to do with stolen AWS credentials being used for crypto currency mining.

Yes, this might also be a possibility

Doubtful, if you're using stole credentials you don't need to care about efficiency and you can just use whatever instances are available to the account.

They're likely doing it as a way to control demand spikes because they have such a smaller pool of GPU instances to go around. Also, it prevents Joe Schmo from wasting one to run a web server because he doesn't understand performance and "just picked the best one".

Interesting - we started launching GPU workloads recently without bumping into that. Might be limited to VPC-only?

I guess if you are a regular user (and have paid some invoices already) the limits will be adjusted automatically

If you've previously asked for a limit increase, it may be grouped with the other server types - I notice that when the limits are increased, all server types increase together.

I can start g2 but not p2 instances (limit 0). My AWS usage is very low, but Amazon has charged by credit card a few times so I guess I am a paying customer (not using free tier)

My first thought: "I wonder what the economics are like, re: cryptocurrency mining?"

My second thought: "I wonder if Amazon use their 'idle' capacity to mine cryptocurrency?"

With respect to my second thought, at their scale, and at the cost they'd be paying for electricity, it could quite possibly be a good hedge.

They sell their idle capacity at spot prices. I am really happy they have upgraded the GPU instances

As far as I understand it, the only way to profitably mine bitcoin is to have super cheap power and ASICs specific to bitcoin mining. Essentially, GPUs won't cut it because you're competing with Chinese miners using ASICs attached almost directly to dirty, cheap electricity generation.

You are right about Bitcoin specifically, but there is a second market for altcoins which use different PoW algorithms [0]. For a lot of these ASICs are either not on the market, or not economically feasible, in which cases GPUs still dominate.

But most of these markets are very small compared to Bitcoin.

[0] https://whattomine.com/

He didn't say bitcoin.

You don't understand it.

Mine other cryptocurrencies that are profitable, or may become profitable. It is VERY, as in EXTREMELY, profitable to mine large stakes in cryptocurrencies that the market hasn't discovered yet.

Especially ones that aren't even listed on an exchange yet and mining is the only way to earn it.

The Darkcoin (now renamed to the politically non-threatening name DASH) instamine was a notoriously profitable use of GPU clusters on AWS.

Don't claim I don't understand it when I was clearly talking about the economics of bitcoin.

Also, if you are paying to mine currencies that don't have an exchange value yet, you are just speculating. The mining isn't profitable at that point because the currency has no value. It's just as likely that you chose to mine a dud. It's exactly the same thing as opening a copper mine at a loss and hoarding the concentrate hoping that it will gain value.

> you are just speculating

you say that like its a bad thing

it is a very profitable endeavor

FWIW it appears that essentially all the Chinese mining is now done with "standed" hydroelectricity out in the middle of nowhere; even coal is too expensive to be competitive.

Fuck bitcoins. I want to know how many electric sheep [1] this beast can breed per second!

[1] http://www.electricsheep.org/

On one hand, normal GPU mining is practically never efficient with respect to the cost of electricity. Even in a hyper-efficient datacenter, Bitcoin mining is now dominated by thousands of ASIC miners who are running hardware a hundredth the cost of GPUs at 10x+ performance.

But on the other hand, these machines aren't just sitting there powered off when they aren't provisioned. They have to keep them warm so they provision ultra quick.

So maybe there is some level of mining they could accomplish with their "idle" cycles, given the efficiency of the datacenter, the power of their GPUs, and the price of Bitcoin, which would be profitable. I don't know.

Can use other types of coins where gpu is still ok.

This would work well for eCoin...

As others have pointed out there are still non-btc crypto-currencies that are worth GPU or even CPU mining[1].

But they don't even have to be making a raw profit vs the electricity they are consuming. If, as you say, the spare capacity must be kept warm anyway, then they only need to be profitable vs the delta of electricity consumption between idle and fans blowing 100%.

So whatever capacity is left beyond the fixed and spot market _could_ still be earning them something.

[1] http://cryptomining-blog.com/6090-crypto-coins-to-check-out-...

Edit: fixed spelling.

This is great - we'll try to get our Tensorflow and Caffe AMI repo updated soon: https://github.com/RealScout/deep-learning-images

How do Amazon (or any other cloud provider) make sure they have enough of these machines to cope with the demand for them without getting too many?

You pretty much answered your own question. First, they have 'too many', second, if they dont, they oversubscribe, third, after running your business for a while, you can do some math on your stats and predict future demand with accuracy that's 'good enough'.

I don't think you can oversubscribe GPUs

But at the price they're charging, if demand exists, they just call nVidia and order another batch of cards

And fortunately, they have enough computing power to do the math.

They lowball with the offer (#instances) and have price regulate the demand side.

After a few months they can estimate the demand and increase supply accordingly (iff profitable)

A lot of people will use spot instances to get cheaper computer power for non-critical calculations. So they can control demand by the price for spot instances.

Also, they'll get a lot of instances up and running before publishing the post. At the moment, p2 spot prices seem very low (~9c for p2.xlarge).

Especially in the GPU space, a lot of calculations will likely not be very time-sensitive. The spot market is optimal to control demand and ensure high utilization.

They estimate, sometimes well sometimes not. Recently Ive seen a region that ran out of an instance type, launching gives you an error, and you're sol for days if you can't adjust. Reserved instances are more than just a price reduction

This is what I'm currently getting in Ireland:

Launch Failed We currently do not have sufficient p2.16xlarge capacity in zones with support for 'gp2' volumes. Our system will be working on provisioning additional capacity.

Capacity planning is a pretty advanced mathematical science, I'm sure there are many Ph.Ds, who are experts in their field, who do this job across the industry.

According to [1], the K80 GPUs have the following specs:

    Chips: 2× GK210
    Thread processors: 4992 (total)
    Base clock: 560 MHz
    Max Boost: 875 MHz
    Memory Size: 2× 12288
      Clock: 5000
      Bus type: GDDR5
      Bus width: 2× 384
      Bandwidth:  2× 240 GB/s    
    Single precision: 5591–8736 GFLOPS (MAD or FMA)
    Double precision:  1864–2912 GFLOPS (FMA)
    CUDA compute ability: 3.7

Is that a good deal for $1/hour? (I'm not sure if a p2.large instance corresponds to use of one K80 or half of it)

How much would it cost to "train" ImageNet using such instances? Or perhaps another standard DDN task for which the data is openly available?

______

[1] https://en.wikipedia.org/wiki/Nvidia_Tesla#cite_ref-19

Priced this config (or close enough) on http://www.thinkmate.com/system/gpx-xt24-2460v3-8gpu

Comes to just under $50,000 for the server or roughly 4.5 months @ $14.40

That's months of continuous use. If you have periods when you don't use it, then you'd have to take the occupation percentage into consideration. It's pretty wasteful not to use a 50K$ server at maximum.

Also, doesn't include electricity, cooling, data center space, all of which might be significant?

Sounds like a great way to build a custom render farm. My home comouter has a dirt cheap GPU but it works well enough for basic modeling & animation. Terrible for rendering, though. I've been thinking of using ECS to build a cluster of renderers for Maya that I can spin up when needed and scale to the appropriate size. I don't know for certain if it's cheaper than going with a service, but it sounds like it is(render farm subscriptions cost hundreds), and I would get complete control over the software being used. I am glad to hear that Amazon is doing this. Granted, I'm more of a hobbyist in this arena, so maybe it wouldn't work for someone more serious about creating graphics.

It is interesting to compare this to NVidia's DGX-1 system. That server is based on the new Tesla P100 and uses NVLink rather than PCIe (about 10x faster). It boasts about 170 Tflops vs the p2.16xlarge's 64 Tflops. If you run the p2.16xlarge full time for a year it would cost about the same as buying a DGX-1. Presumably Amazon releases their GPU instances on older hardware for cost savings.

http://www.nvidia.com/object/deep-learning-system.html

Stupid question: are GPU safe to be shared by two tenants in a datacenter? I read previously that there are very few security mechanisms in GPU, in particular that the memory is full of interesting garbage. So I would assume no hardware enforced separation between the VMs too.

The GPUs aren't shared between VMs, each GPU is dedicated to the VM it is allocated to. The GPU memory gets wiped on initialisation, just like host memory, so there's no leakage between successive VMs allocated the same hardware.

If you do absolutely care about not being multitennanted, you can stipulate exclusive hw allocation when requesting the VM.

> The GPU memory gets wiped on initialisation

Well, an attempt is made. As we can see in the case of WebGL, there are a multitude of interesting effects.

Can't speak to what happens on a Windows box, but across AWS instances it's definitely the case.

At the sort of scale these instances run I'd be surprised if you're sharing any physical hardware at all, my guess is that they provision a single VM on each GPU instance.

That's not mentioned in the article, did you work on this?

Disclosure: I work on Google Cloud.

Nvidia has been doing a lot of work on hardware virtualization of GPUs. A goal of which is to run dozens of low-requirements desktops (think users running MS Office in separate Windows VMs) off of a single high-end server with a single GPU.

http://www.nvidia.com/object/grid-technology.html

Pricing?

https://aws.amazon.com/ec2/instance-types/p2/

From 0.90-14.40 USD/hr.

Yess thank you, thank you, thank you. I was just signing up for the Azure N Series preview but we're good to go now :).

Anyone knows if video transcoding on GPUs (with FFMPEG) is viable nowadays? If yes, what are the gains?

They have chips dedicated to video encoding/decoding with the latest cards doing h265. Nvidia has an ffmpeg patch here https://developer.nvidia.com/ffmpeg

Decoding is good but hw encoding usually has very bad quality/size ratio compared to best software encoders.

This depends on if you're doing live transcoding or VOD transcoding.

I work with live transcode, and it can be beneficial to run on g2 instances. Running c4.4xlarge I can transcode a good number of 1080@30 in with 1080/720/480/360/240@30 out. With a proper g2 instance I can transcode more simultaneously.

So cost efficiency really depends on sustained traffic levels. I scale out currently using haproxy and custom code that monitors my pool and scales appropriately. But I monitor sustained traffic levels to know when it makes financial sense to scale up.

If your main concern is transcode speed, CPU is likely sufficient -- I am unable to transcode faster than real time with live transcode.

> I am unable to transcode faster than real time with live transcode

Well, yeah. Maybe quantum computing will change that one day!

What is it you're working on? Colour me intrigued.

I'm not sure if you're kidding or not, but just in case: it is desirable to be able to encode faster than real time (ie: 1 hour of media takes less than 1 hour of time to transcode) so you have leftover processing cycles to do other work.

I know I know aye, the way I read initially amused me for a moment before my brain clicked into gear :)

This is true. But real time encode is essentially what you need to do with live video instead of VOD transcode. You have other processes doing the other necessary work during the transcode.

All in all, I look for somewhere around 0.25s to complete the rest of the work and deliver the segment to CDN.

I'm doing live video ingestion. Transcode to DASH/HLS at 5 different qualities + alternate audio/subtitles where provided.

Damn, I would love to have an excuse to play around with this.

Reply on news.ycombinator.com