Comment by jiggawatts | Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Parent Full thread

I'm not an economist, but you touched on an observation that I've made that I can't quite articulate using the appropriate technical terminology:

When products are not fully vertically integrated, and there's a large markup dowstream, the upstream manufacturing is highly incentivesed to "penny pinch" in a way that would be perceived to be highly counter-productive in an integrated organisation.

The example that first made me notice this was power management in server systems. If you're running $100K worth DBMS software (licenses) on a $100K box, all of which has a further $10-100M of web apps and systems relying on it, then it would be madness to throttle that CPU to save $50 in electricity annually, right?

Well, guess what: most places do this! It happens because they outsource the low-margin upstream component (colocation/hosting/cloud), and when you're a data centre provider living on thin margins, the cost of power / cooling can be very substantial relative to your own costs. If you rent the rack space for that server out for $500 per annum, that $50 of electricity is 10% more in your pocket! That's a big deal!

Meanwhile, that $50 saved has slowed down $100M worth of stuff, sometimes by 50% or more. But... that's not the hosting provider's problem.

(PS: Azure does this, and they don't let you turn it off.)

> that $50 saved has slowed down $100M worth of stuff, sometimes by 50% or more. But... that's not the hosting provider's problem

Provided their clients don't figure out what's going on and move to another provider. If the client has millions of dollars riding on that infrastructure, they'd likely be doing enough performance-monitoring to notice this kind of thing, no?

> Azure does this, and they don't let you turn it off

Azure really don't offer any never-throttled instances? What about their dedicated hosts?

> Provided their clients don't figure out what's going on

I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly.

Even if cloud vendors do fix this one thing, there's about a dozen more such issues where the suppliers' interests don't line up with the downstream consumers' interests.

> Azure really don't offer any never-throttled instances? What about their dedicated hosts?

I don't believe so. For example, their highest performing single node is the HBv3, with 2x AMD EPYC 7V13 processors for a total of 120 cores / 240 threads.

On this page: https://docs.microsoft.com/en-us/azure/virtual-machines/work...

There is a box that says:

    Nodes per Socket (NPS) = 2
    L3 as NUMA = Disabled
    NUMA domains within VM OS = 4
    C-states = Enabled

The technical name for CPU power management is called "C-states". In other words, they've left it on the default (enabled) on a machine that they charge $40K/year for!

> I'm always pleasantly surprised when I see power management correctly configured at a client precisely because it is so rarely done correctly.

Let's say I have Linux instances in the cloud. What would be the steps to take for proper power management?

As far as I know, there are no steps you can take if you're running your instance in a virtual machine. Only the hypervisor host and/or the firmware can control the processor power and sleep states.

TL;DR: C-States improve performance.

I'm not sure calling C-States throttling is completely accurate, at least not in the way that you imply.

Modern CPUs have multiple mechanisms that can be called throttling in some way, but C-States are the least offensive and in modern CPUs actually increase performance.

There's a good basic explanation here [1] but in short :

- C-States is about idle cores, and powering them down. You can actually configure how deep you can power them down. Waking up does have a tiny cost in latency, yes, but it's mostly irrelevant (your core spends most of it's time waiting for data).

- P-States are what I would call "ok" throttling, those happen when non-idle, modern cores can scale their frequency up and down based on how much workload they have. Again, in theory purely about power saving and close to 0 impact on performance.

This was simple, and used to be just about power saving, disabling was ok. But things changed massively about 10+ years ago with the introductions of various Turbo mechanisms.

Turbo is about looking at your CPU as a whole (and not per core), and maximising performance of the loaded cores by pushing their frequency, using the power budget that used to be allocated to the idle cores (that are now powered off).

The way its done is by adding a few extra P-states that push the frequency a few hundred MHz. The more "C-State-down" cores you have, the higher the frequency of the working cores is pushed.

Extreme example : you have a 28 cores CPUs running a single thread workload on one core only, with 27 idle cores. With C-States disabled, you lose Turbo and are stuck with the single core running at the "non-Turbo" frequency. With C-States enabled, because you idled the non used cores, your working core gets higher clock speeds and higher performances.

(Things get more complex with AVX2 because those units have a large power consumption, and this is where you get into the very bad throttling)

[1] : https://software.intel.com/content/www/us/en/develop/blogs/c...

This might be the case for some artificial benchmark, but I have never seen such a case in practice.

I have run many, many tests on real-world workloads, not artificial 100% loads of either a single core or all-cores, as you typically get in benchmarks. In all cases, disabling C-states made overall performance at least 20% better, and often 200% better (3x faster)!

There's lots of reasons for this, and I could write pages and pages on how the jitter introduced by the sleeps is murder on TCP throughput, or how the frequency ramp up time is glacially slow, or how most servers are idle most of the time and only the cold-start latency matters to end-users. Then I could keep ranting about the waste of it all, and go on and on about how Californian greenie hippie morons tried to save the planet one database server at a time. Then finish up by showing that 10 Gbps Ethernet's latencies interact with the core sleep timings just so to make things as bad as possible.

But instead of taking my word for it, you could just do your own tests on your own software, ideally in a production system involving n-tier networking instead of some isolated, synthetic benchmark.

PS: By design, power management has no impact on anything running at 100% load for extended periods, so no synthetic benchmark will ever show any impact. That's precisely why very few people are aware of the issue...

Can confirm. I've seen C-states absolutely wreck performance on certain low latency workloads. In particular, lots of small random writes on a storage cluster.

Unfortunately it turns out that "low latency workloads" are not some esoteric thing only high frequency traders care about, but is an accurate description of most enterprise software. Anything n-tier, anything with an API tier or an external cache, anything vaguely K8s-like, or even just a database and a chatty ORM.

> Provided their clients don't figure out what's going on

As far as I can tell, very few people ask similar questions. Those who do, come to interesting conclusions: https://jan.rychter.com/enblog/cloud-server-cpu-performance-...

It's so rare to see cloud benchmarks! I'm surprised this guy hasn't had a scary takedown letter from some lawyers yet...

"There. I've said it. These [Xeon] processors are slow. Dog slow, in fact."

My gimmick recently has been to upgrade Intel CPU VMs to AMD VMs. One benefit of the cloud is that this is literally just a menu selection. If it was on-prem, it would be a one-month argument to convince people that it's risk free, then a one-month argument with some beancounter, then a month of waiting while the pallets of kit ship in from Asia, then another two months of rack mounting and prepping for production.

"They let them pack lots of slow cores onto a single CPU die, put that into a server, and then sell twice that number of cores as "vCPUs" to us."

Azure also charges for licenses on a per vCPU basis as if* they were physical CPUs! This means that in effect they've doubled their Windows and SQL revenue stream.

Hmm,

I agree with your overall description - I believe economist describe problems of this sort as externalities[1].

The example of externalities that comes to mind is shipping companies running absurdly large ships to shave margins and so being concerned that their behemoth might lodge in the Suez Canal and disrupt much of world trade.

IE, producers not being concerned about the larger impact of their decisions.

But your example leaves me scratching my head. My impression is that cloud providers effectively selling CPU cycles by the dollar. There must be some pricing tradeoff for fast versus slow cycles. If a cloud provider can sell 20 throttled instances cheaper than 17 unthrottled instances, that's a deal that might suit buyer and seller.

But maybe there's something I'd missing here.

[1] https://en.wikipedia.org/wiki/Externality

Reply on news.ycombinator.com