I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month, if not every week...
AWS does not have a 100% similar VM, but you could have something close for ~ 20,000 USD monthly. Not that bad.
However, storage costs alone would be astronomic. Like > 100,000 USD / month.
I have no idea how much outbound traffic Let's Encrypt serves, but that also could be a quite relevant expense.
OFC I also don't know how much Let's Encrypt pays for energy, cooling, operations, real estate, etc... but:
> I bet that compared to an equivalent load hosted on AWS, that lovely box pays for itself in full every month
I would not take the other side on that bet
24x 6.4TB Intel P4610 NVMe SSD = 24 x $310 = $7440
2x AMD EPYC 7542 = 2 x $1300 = $2600
2 TB DDR4 ECC RAM ~ $13700 (estimate from a couple of Google results)
One thing to note is that a VAR (as mentioned elsewhere) will knock 75% off the price listed on Dell’s website.
Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
Even on Newegg, SAS SSD of that size are $900-2000, so add warranty and service on top of that.
Makes sense.
> Another this is that that is way too cheap for those SSDs. Enterprise SAS (not plain SATA) SSDs are a lot more than $310. Our 7.68TB drives are about $2k each, but worth it if they stay problem free.
I was able to find these two enterprise-grade NVMe SSDs on Newegg:
https://www.newegg.com/p/2U3-0005-000J8 ($474, 7.68 TB, 1 DWPD)
https://www.newegg.com/p/2U3-000S-000Y8 ($360, 6.4 TB, 3 DWPD)
This is something more similar to what is in those Dell servers (and there are 24 of them):
https://www.newegg.com/samsung-pm1643a-7-68tb/p/2U3-0005-000...
https://www.newegg.com/p/0N7-0133-00003
There is certainly a markup with Dell, but it’s sort of like a cloud vendor - pay for the warranty and service, and be (somewhat) hands off if something breaks.
that's the one of points of article: Epic cpus have 128 lanes for a while, and that's how they upgraded to 24 NVMEs.
I don't know all the ins and outs of SAS vs NVMe. Maybe someone else can chime in. I am at the end of my knowledge now.
I suppose one benefit is the availability of hardware RAID controllers, as hinted in the article. But it does seem interesting that NVMe is cheaper than SAS, while theoretically having higher bandwidth.
My personal workstation at the side of my desk has 6 pcie nvme ssd's and I can add 4 more without breaking the lane bank.
> Makes sense.
Does it? Adding a middleman that needs to take a cut to exist reducing the price makes sense to you?
(I work at Let’s Encrypt)
If Amazon looks reasonable for something like this, the math is wrong. They’re renting boxes at 60-70% margin.
does it have similar instance in principle: you rent dedicated server with lots of ssd attached and with no fear that instance will be stopped any moment for whatever reason?..
Is that the on-demand cost, or the reserved cost? For comparing to buying a server outright, you should be comparing the reserved cost. I’m not sure exactly which instances you’re looking at to get $20k/mo, but I see some instances with 64-128 cores/1-2 TB memory for <10k/month.
For storage, I’m not sure how you’re getting >100k… I plugged in the highest IOPS I could for io2 volumes for 150 TB of storage and got 30k/mo. Also worth considering here that you don’t have to provision all 150 TB up front - you could start with 5 TB and increase in size as you grow, for example.
Still gonna be hella expensive but all of this changes the calculus quite a bit from your estimates.
- ED of ISRG / Let's Encrypt
I spec’d a current gen server with:
- 2x more cores
- 50% more RAM
- 50% more storage
For only ~$80k, and that’s not even with a discount and is a more powerful current gen server with more cores/ram/storage.
It’s $41k if I matched the specs exact (but that’s a bit unfair to do because the Dell server is 2-3 years old).
Never understood why people are so infatuated with "cloud" options. Yes, it is convinient, but you are absolutely paying at least order of magnitude more for the same amount of compute/storage.
On AWS reserved instances with 8x A100s or H100s (if you can even get them) cost more per year than the total upfront retail price of the equivalent pods from Lambda Labs. The on-demand price is even more absurd.
Once you have the bandwidth at your location and you don't need to be present in multiple locations, it's cheaper to self-host.
Next step would be colocation, but for a start, using cloud offerings is a cheap way to be a part of the internet.
You can get "baremetal"/dedicated servers from places like Hetzner and OVH that give unmetered gigabit connections for like $50.
Additionally, if you are actually pushing close to that much traffic, you can negotiate guaranteed commit prices w/ AWS that are competitive (especially when you consider the quality of bandwidth. I can only get ~100 mb/s to my hetzner server because of how bad their peering is. I can easily saturate my 1GB connection to anywhere in AWS.)
---
(Having said that, this does only apply to egress via cloudfront. Things like charging for intra A-Z bandwidth within the same region is insane, and for many workloads may be surprisingly expensive.)
looks like your information is obsolete by 5 years: https://www.hetzner.com/news/traffic-limit
Their actual docs: https://docs.hetzner.com/robot/general/traffic/
And reports of throttling once you actually hit certain limits: https://lowendtalk.com/discussion/180504/hetzner-traffic-use...
Other offerings (e.g. 20gb/s uplink) and small cloud servers have traffic limitations.
But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
No, you clearly stated that 1gb/s servers have 20tb cap, but no such products have such limitation, no links say anything like that.
It makes total sense to cap 20gb/s, because such unlimited bandwidth is clearly too much.
> But more importantly, the second link I posted shows how that despite being "unlimited" - it's not uncommon for hetzner to throttle / close your account if you go over unstated traffic limits.
you can argue that, though they didn't close or throttle account, they sent warning, and cap was 12 times higher than what you previously stated.
While cloud overcharges for McD product, I know what I get.
That wouldn't happen if the internal bureaucracy for firing the laggards wasn't so excruciatingly slow.
A datacenter makes sense if your usage profile is steady state and hardly ever changes, or if growth rate is predictable and capacity can be procured in advance. Any other use case is better suited for the cloud, IMO.
And it costs like 1/5th of cost of “cloud”, and the later still gets you “the cloud”.
Now you’re in the business of maintaining/patching/imaging/securing your operating system and all of the libraries/builtins it comes with...along with your app's stack.
That's not nearly as simple as "Hey, here's my code, Lightsail; ship it and present me its URL. Thanks!"
that looks like some marketing material without any specifics..
A ton of people forget that a bunch of servers across a few colocations can pay for itself in months, especially if you go for second-hand gear that is dirt cheap.
Again, going (back) to colocating hardware could not be good fit. But with modern management tools and datacenter services like 'remote hands' I think people should not reject it upfront.