Hacker News Reader

Comment by bpodgursky | Hacker News Reader

None of this math is correct. You absolutely cannot get 128 cores for $500. Please link to a single valid-looking eBay offer at that amount (you may have seen 128GB and thought it was 128 cores). I see 32 cores for $500, at best.

Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo, not well over 10k.

Third, you really underestimate electricity costs if you're running a server 24/7 at all cores.

Last, and most importantly, most people simply don't run servers 24/7. They run batch computations for five hours a week, or a day, and then spend a week or two crunching the numbers.

There's some case to make that for certain institutions with very particular compute needs, running on-premise might make short-term financial sense (let's not even get started on labor costs). But it's really inaccurate to call cloud computing a "rip-off" for anything but the niche-est customers.

View on HN

> I see 32 cores for $500, at best. Second, 640 cores on an m4.large for 3 years upfront (the equivalent purchase) is $8755/mo,

You can beat that cores per dollar handedly that if you're willing to use sufficiently odd hardware.

E.g. my Intel cluster here is mostly I have systems with 4x E7-8880v4 (22c@2.2GHz). I paid $150/chip + $450 host (thanks ebay) excluding ram.

(I have 8 such hosts now, so 704 cores)

So essentially one months cost for your AWS price w/ a 3year commit pays for an equivalent core count in systems for me... Throw in another month for the ram. Round up to another another to count for the (quite) non-trivial power usage.

You do not have to be anywhere near 100% utilization for AWS to be an extremely poor value large computing jobs compared to thrifty spending on the surplus market.

how?

>I paid $150/chip + $450 host (thanks ebay) excluding ram.

E7-8880v4 are $600 used each. Quad core mobos are not cheap, ranked ram is not cheap. Even if you pulled these from a vendor going out of business the bit coin mining crews would suck these up in heartbeat.

Where are you getting your prices?

Ebay. Hunt harder (and best offer harder). :) E7 CPUs sell absurdly cheap at times because it can be a bit hard to find compatible systems.

(similarly, quad socket systems with slow or no cpus go cheap from people that don't know much about them)

> ranked ram is not cheap.

Yes, ram for quad socket hosts it adds up, since you need at least 2-8 dimms per socket for full memory bandwidth (which was why I explicitly pointed out that I left it out).

On the plus side, those systems tend to have a lot of dimm sockets so you can use lower capacity dimms.

> bit coin mining crews

nah, not useful for that.

On a side note, you would expect people to be super excited to explore possibilities of new infrastructure models. Instead, most posts related to lambda, etc... always come with this sort of reply.

For my use case (infrequent, easily separable jobs) FaaS stuff is a godsend. In the last year I've been doing things that would be orders of magnitude harder and more expensive thanks to it.

> Third, you really underestimate electricity costs if you're running a server 24/7 at all cores.

not to mention the cooling, or the per square foot cost of renting the space for this hardware. anything close to what's being described here is going to need some sound deadening (be it distance or material) from humans.

Frankly, I think many people here are overestimating the cost of electricity and cooling. My company leases several 42U cabinets with 208V 30A circuits in a modern SOC 2 Type 2 audited datacenter. We pay $600 per month per cabinet. Excludes IP transit costs. Granted this is in the Pacific Northwest where we have plentiful cheap hydro power.

> We pay $600 per month per cabinet.

Still a world of difference between AWS's 10k/month cost to access that many cores.

Comparing a Dell r740 (56 cores, 192GB ram, 1.6 kw psu), you need 11 of these for 640 cores (for a comparison with [0]). They're roughly 2500 each, so your up front hardware costs are 27,500.

Assuming the $600 covers power, rack costs and cooling, your data center bill is 6600/month.

For three years, the AWS bill is $315k, and the self hosted option is $265,100 (best case). I'm not up to speed with the pricing trends to say whether the gap will change over those 3 years based on data center changes or aws changes, but the difference is ~50k for three years, or roughly 3 months of an engineers salary over the cost of three months.

[0] https://news.ycombinator.com/item?id=23514698

Did you consider that AWS counts threads, not cores?

No, I did napkin math that compared an unverified number farther up the thread to the first server I found on Google. If you're considering buying hardware you should do your own numbers, I was merely pointing out that difference isn't 10k/month vs 27k up front

> so your up front hardware costs are 27,500.

That's about 3 months worth of AWS to access a similar amount of computational resources.

And if you don’t need them all the time , instead of 3 years RI you can us spot fleet or turn off half of the servers at non peak hour.

It's $600 per cabinet, not $600 per server. If you go by the PSU, that's 3 cabinets. But that 1.6kW PSU is quite overspecced for multiple reasons, like efficiency and the fact that it derates to half on 110v power. 28 core xeons have a TDP in the 165-255 watt range, after all. So even 1kW per server is overkill and that's only 2 cabinets. $1200 per month, not $6600.

With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years, which is pretty much the salary of one person to look after that system for the lifetime. I'm not arguing that AWS is good value, im arguing that the upfront cost of purchasing the hardware isn't comparable to reserving EC2 instances up front.

You still need someone to look after your systems if you're using AWS. That's not something special to buying your own server.

And management of the hardware itself doesn't take an extra employee. It probably averages to less than one workday per month.

> With your numbers, that puts the total running cost at 71k, or a saving of $245k over 3 years

To compare apples to apples, that hypothetical pricetag buys you the system's max capacity for the entire period. In AWS, the invoice from using any of their serverless offerings grows wildly even when utilization stays way below the resources of that hardware bundle.

For instance, let's keep in mind that AWS charges:

* API Gateway per million of requests,

* Lambdas per timememory used by a single call, including the time it wasted with hot and cold starts,

SQS per queued message,

* WAF per request through the firewall,

* Etc.

> which is pretty much the salary of one person to look after that system for the lifetime.

My average time spend in data centers when handling racks in two different location was around 3-4 days a year. Buying on-call, out-of-hours support is going to cost you a tiny fraction of a persons salary per year, either on retainer and/or based on hefty day rates.

Hardware failures simply do not occur often enough for a rack or two worth of hardware to typically require enough maintenance to make it that expensive, and remote hands (people who act based on a ticket) are available in pretty much any data centre at low costs. If you get servers with IPMI etc. you typically pop in to wire up or retire equipment and dealing with the occasional failure.

Source: Have provided those kind of services for years.

Reply on news.ycombinator.com