Hacker News Reader

Comment by aftbit | Hacker News Reader

It's very interesting to see this. I wonder what Amazon, Facebook, Google, and Microsoft are going to do in the next few generations. Will they be moving to 2U1N as well? Or do they have liquid cooling in mind? Or maybe they'll be keeping lower power budgets? Or building data centers that run cooler and support higher power density (probably not great for the environment).

I knew fan power usage could be a substantial amount of a server's power draw, but I didn't realize they would need 320W to move enough air to cool 500W of dissapation.

View on HN

Yes it's insane to waste 70% of the CPU power again on cooling the CPU (read: cutting into power density).

Oxide's rack uses 80mm fans that run at less than half the stock minimum speed to reduce this waste and run super quiet: https://oxide.computer/blog/the-cloud-computer

It's nuts how much energy server fans can use.

I recently picked up an R730xd. By default, a non-Dell-branded PCI-E device will force the fans to ~16000RPM, but there's an IPMI command to disable that and go back to regular, temperature-driven, fan control. Total server power came down by 70W after issuing that command.

This is very uninteresting and way behind the big cloud vendors high density installs. Theyre just saying “when power/cooling limited use a form factor that optimizes your remaining constraints, like RU.” So yes, 2p & 3p edge sites are commonly limited to 5-10kva, and you end up with low density racks and “extra” physical space you can use. In the modern dense 1p deployments youll see 20kva, or more, per position. The physical form factor always follows those constraints based on cooling/power/connectivity/weight/etc.

Even if you had unlimited power per rack you wouldn't want to waste it on fans.

I thought current rack standard installs were like 40kW per rack, with high end all-AI type racks being like 100kW each?

Where the heck are you seeing 5-10kva?

OVH is doing liquid cooling to some extent since 2003. https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...

The latest Zen4c-based EPYC chips are reaching very high power densities. They're reportedly targeted towards hyperscalers that are badly cramped for space in their datacenters but can be more flexible wrt. power budgets. In a way, these concerns had to be there already with HPC hardware, so there's probably well-established approaches for dealing with them too.

The major cloud vendors have their own data centers & do not need to deal with space or power limits at a rack unit. They have the entire data center worth of space and power supply to work with.

If the data centre is owned by digital reality, equinix, telehouse, etc, is it really theirs? https://aws.amazon.com/directconnect/locations/

Direct Connect locations are not the locations where the main region data centers are. AWS has a network link from their DC to Equinix, etc to make it easier for customers who are in Equinix's colo to make a direct connect to the AWS network.

All data centers have space and power/cooling limits. If you're in someone else's DC, they are usually going to quote you as per-rack power, but if you really want you can probably convince them to let you use 2 racks worth of power in one rack and hold a second rack empty. In owned and operated DCs, it's kind of similar, if the DC was built a while ago, as server power density has increased, a significant portion of the floor space may be unusable for racks.

10-20 years ago, 1U and blades made a lot of sense because space was a major constraint; now 2U or larger can make sense because 1U has a lot of tradeoffs for size, and power is the bigger constraint.

If space density is still needed, 2 nodes in 2u or similar narrow but tall configurations provide a more square profile that allows generating airflow with larger fans which is more efficient than the 40mm fans needed in 1U systems.

Reply on news.ycombinator.com