Hacker News Reader

Hacker News

Comment by pid-1 | Hacker News Reader

Parent Full thread

pid-1

I did some napkin math (could be very, very wrong) but this server costs ~ 225,000 USD according to Dell's webpage.

AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.

However, storage costs alone would be astronomic. Like > 100,000 USD / month.

I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense.

OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but:

> I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month

I would not take the other side on that bet

View on HN

$225K for this sounds bonkers:

  24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440
  2x AMD EPYC 7542 = 2 x $1300 = $2600
  2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results)

Those add up to something like $25K. Sure, there's also the price of the motherboard, chassis, maybe some other peripherals like external network cards, assembly + support + warranty etc. but that doesn't explain an 800% markup.

We recently purchased a similar server.

One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.

Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.

Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.

> One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.

Makes sense.

> Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.

I was able to find these two enterprise-grade NVMe SSDs on Newegg:

  https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1 DWPD)
  https://www.newegg.com/p/2U3-000S-000Y8 ($360, 6.4 TB, 3 DWPD)

Is there some kind of catch I'm missing?

I am not too much of an expert on enterprise hardware, but those are PCIe interface. I don’t know how possible it is to rack up 24 of those in a single server (you would run out of lanes).

This is something more similar to what is in those Dell servers (and there are 24 of them):

https://www.newegg.com/samsung-pm1643a-7-68tb/p/2U3-0005-000...

https://www.newegg.com/p/0N7-0133-00003

There is certainly a markup with Dell, but it’s sort of like a cloud vendor - pay for the warranty and service, and be (somewhat) hands off if something breaks.

> I don’t know how possible it is to rack up 24 of those in a single server

that's the one of points of article: Epic cpus have 128 lanes for a while, and that's how they upgraded to 24 NVMEs.

Oh yeah, the article. I guess this thread got sidetracked on the topic of Dell's pricing :). I wonder how common a 24-drive NVMe server is.

I don't know all the ins and outs of SAS vs NVMe. Maybe someone else can chime in. I am at the end of my knowledge now.

I suppose one benefit is the availability of hardware RAID controllers, as hinted in the article. But it does seem interesting that NVMe is cheaper than SAS, while theoretically having higher bandwidth.

Yeah, I feel SAS is obsolete tech, and will be replaced by NVMe everywhere going forward.

Threadripper and Epyc has been smashing the pcie lane limit for a while now. That's why Epyc is kicking intel's ass in server applications.

My personal workstation at the side of my desk has 6 pcie nvme ssd's and I can add 4 more without breaking the lane bank.

> > One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.

> Makes sense.

Does it? Adding a middleman that needs to take a cut to exist reducing the price makes sense to you?

The hardware may have cost more in 2021 when this article was written.

For a critical system, you should really have two, for HA purposes. Do they?

Yes. Let’s Encrypt has two locations, each of which has fully redundant hardware, so that’s a minimum of 4. We actually have a few more.

(I work at Let’s Encrypt)

This did not cost us $225k. About half that. Nobody pays the website price, you pay a lot less via a VAR.

- ED of ISRG / Let's Encrypt

Nobody is paying base price for a box like that. I’d probably bid them against HPE and pay ~90-100k.

If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.

> These are expensive servers, crossing into six digits, but not $200k.

https://news.ycombinator.com/item?id=25865967

It’s closer to $80k, if you went via a VAR.

I spec’d a current gen server with:

- 2x more cores

- 50% more RAM

- 50% more storage

For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage.

It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old).

https://www.siliconmechanics.com/system/rackform-a335.v9

> AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.

Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month.

For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example.

Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.

They're also using ZFS in Raid1+0, so 38.4TB of usable storage. $14,566/month on AWS io2 with max IOPS.

I'd also bet money that this has higher disk I/O performance than the rough equivalent on Amazon.

> AWS does not have a 100% similar VM

does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..

Reply on news.ycombinator.com