An Inferno on the Head of a Pin
blog.codinghorror.com
blog.codinghorror.com
In the past I'd assembled several PCs and my less-than-rigorous setup process typically involved slotting the CPU, RAM, and other cards, then plugging it into a monitor and powering up to run through a quick POST just to make sure nothing was misaligned or improperly seated. Then I'd power off, put down some thermal paste, mount the heatsink/fan, and connect any remaining cables for drives, etc.
Well, in the past, this was fine. Sure, I wouldn't want to run things without at least the crappy little Intel or AMD CPU fan but 5 or 10 seconds was no big deal. But this time I was building for a friend who had me order one of those fancy new 1GHz Athlon chips. I was psyched as well because the small markup I'd added for my time helped me afford my own fancy 1GHz CPU and new motherboard for my own PC.
So I followed my usual procedure and slotted everything for the initial POST (nothing worse than getting everything connected in one of those old PC cases with IDE cables and wires everywhere just to have to unhook it all because something wasn't seated properly) and powered on.
Blink.
Nothing.
Hmm...Did I not get the CPU seated properly? Let's pop it out and reseat it. OWW! Burned the crap out of my finger and got a blister!
Uh oh. It sorta smells like the magic smoke even though I don't see any. Well, eventually I realized what had happened and learned that even a few seconds without a heatsink/fan had been enough to fry that brand new 1GHz Athlon.
Thankfully I had the one I'd ordered for myself and I was able to finish the build that my friend had paid for. But it was a rather expensive (for me at the time at least) lesson in the need for proper cooling on these modern CPUs.
I still feel cheated by Intel, because working on that crap was like working on an airplane turbine. The noise and the heat was impossible to tolerate to the point that some companies started to sell solutions for this thing back then, some even using water. I ended up buying a huge blue dissipator.
Intel used me to test their new Hyperthreading technology, the processor was called Prescott, a.k.a "Pres-Hot".
After setting that bad boy up, it ran so hot that my room was about 5 degrees hotter than the rest of the house. I finally started putting it to sleep for most of the day just to keep the heat down.
Generally speaking, no. Some are more tolerant of high temperatures, but that's it. And higher temperatures in general will degrade the processor over time.
Because the Intel CPUs that followed actually had power saving features, previously found only on mobile versions.
Those P4 would just run hot all the time, not pleasant being in computer lab with two dozens of them + CRTs and AC that couldn't keep up.
80286 needed no heat sinks, even 20-MHz models.
80386 used to do well without radiators, too, or with entirely passive cooling, even the faster DX variants.
Frankly, I don't remember low-power i486 (e.g. 40-MHz versions) to have heat sinks, too. You definitely wanted a heat sink, though usually a passive one, on a DX2 (66 MHz) or DX4 (100 MHz).
Before then, I don't remember my IBM 386/75 or 486 needing fans ... but the 486DX2/50 had problems with RF emissions from the corners of motherboard traces (IIRC) so they left external clock speed lower for many moons ...
While trying to get as many of those machines working as possible, I unseated the CPU (can't remember why), and I guessed the orientation wrong when putting it back into the machine. I don't remember seeing or smelling any literal magic smoke, but it must've gotten let out, because that machine never booted again.
Luckily, that wasn't any kind of new hardware. I've got an active imagination though, and it always comes to mind when I am working with new and expensive equipment.
Took the CPU back to the store the next day to complain that it didn't work and it took them about epsilon seconds to figure out what had happened. I bought a new CPU and hastily left the building. They must have been laughing at that for days.
One time I really did think I heard a crunching/grit type sound when mounting a heatsink and my stomach dropped out until I got the system powered on and verified that nothing was broken.
Seriously, these things were terrifying to mount a heatsink on back in the day with the way that thing sticks out, rubber pads be damned: https://i.imgur.com/6w1oVMS.jpg
> I remember cooling the early CPUs with simple heatsinks; no fan. Those days are long gone.
Interestingly, for desktop machines, this is not quite correct... there is still a "fanless" movement going. My roommate has a PC that doesn't have any physical moving components: no fans, no hard drives.
Fanless designs are not space-efficient, though; see this fanless CPU heatsink: https://www.quietpc.com/nof-icepipe which is rated for up to a 95W TDP CPU - enough to run an Intel i7-6700K, which has a TDP rating of 91W.
If you're willing to build a very quiet machine instead of a silent machine, liquid cooling with very quiet fans (that eventually ramp up based on temps) is very workable, and means your machine is effectively silent (i.e. at or around ambient noise levels) 99% of the time.
No graphics card, just the internal GPU of a 4th gen i7 (purchased 2013).
The heatsinks for fanless designs are enormous, but the result (no moving parts in your PC, if you eschew hard drive(s) as well) is really neat in my opinion, and great if you do audio work - as my roommate does. Totally silent, guaranteed.
You definitely need to plan for it, though. The heatsink doesn't offer enough clearance for some of the taller RAM sticks on some motherboards.
The temps aren't all that bad either. If I remember correctly, my roommate's PC operates at temps that are roughly equivalent to the stock Intel heatsink/cooler. Not the best, but given that it doesn't have a fan, pretty nice.
So I invested in a cheap and cheerful Chromebook, put linux on it (Gallium OS is the easiest operating system to install, ever), got the NFS mounts working, got my desktop to proxy serve my dev domains, put 'Synergy' on there (so I can use the same keyboard/mouse on all machines from the Chromebook) and made sure I could also work fully remote with it too (local repo). I also got 'X Windows' to work nicely.
As a glorified terminal my Chromebook has full HD, massive battery life (all day) and zero noise. In fact I wish it was 'warmer' in use, such is the low-end Celeron's feeble and fanless heat output.
It seems to me that laptops actually do not last the distance if they are over a certain size - 15". The thermal management is just not up to it and it is only a matter of time before the fan is running permanently with the CPU throttled. After a couple of dead laptops one thinks 'desktop/server', somehow silent and low power so it can be left on... But this doesn't really exist unless you invest in silent cooling.
With my bargain basement Chromebook I can do everything I want to do although for some things like a time doing graphics work, I will go to the faster machine. This faster machine no longer has lots of cables attached to it, the keyboard/mouse is shared from the Chromebook so it becomes a box with power in, network and HDMI out, neatly tucked out of sight in an adjacent cupboard rather than roaring away on/under my desk.
Another bonus of the Chromebook is that it's lameness is a feature. People have rubbish computers as well as posh ones, I need to test for all devices and what will work on a low end Chromebook will fly along on anything more normal.
> Put the chromebook into 'developer mode', wiping the hard drive
> Physically open the device, breaking the 'warranty void' sticker, in order to remove a write-protect screw
> Change firmware flags to allow booting off of unsigned partitions
> Replace the firmware entirely, risking bricking the device
Apart from that it was easy, and the installation program was faultless. I do use my Chromebook as a dev machine, and I agree with you that the poor performance is more of a help than a hindrance. Code editors don't stress out any remotely modern computer, and if your code runs well on the Chromebook then it should be fine anywhere else; very few machines have worse specs. I think they're fine machines, and GalliumOS is everything one would wish, but I really couldn't count ease of installation amongst their features.
One thing though - sound. This only works on HDMI which again is a feature - I can't procrastinate with videos. Installing 'WinZip' on a PC back in the day when I used 'Windows' was harder and certainly more fraught with danger.
The defining feature of a fanless build really is cost. It's hard to build a machine that will stay silent under load for cheaper than the heatsink I linked above. At the same time, fanless builds don't dissipate enough heat to enable high overclocking (so you can't eke out the same performance as you could using fans).
A free business idea: create decent looking fanless radiator cases for computers. Make them look like minimalistic furniture, not like kid's toys or faux Apple imitations.
1. Mid-range cards and higher consume twice as much as a quad-core CPU.
2. Much more vertically constrained (other expansion cards on the bottom) than a CPU, so heatsink designs are very limited
3. Heatsink is much less standardized, partly because of (2), and partly because of different PCB sizes and the fact that is has to cover VRM chips as well.
There are actually microcontrollers that use around 1milliwatt for very low power applications. For example the msp430. TI have a neat video of one running using power generated from grapes:
http://www.atmel.com/products/microcontrollers/arm/sam-l.asp...
35µA/MHz, so with careful selection of peripherals you probably don't even need a whole milliamp.
I put one of the boards under a microscope and looked at
some of the ancillary parts. There's a 22 uF decoupling
capacitor. No real surprise there. It appears to be one of
those nice Kemet polymer tantalums designed for high-density
SMT applications.
The datasheet pegs the cap's internal leakage at 22 uA. That
means the capacitor draws a thousand times more power than
the dozing CPU.The follow up would be "an Inferno in your Rack".
Also, do you actually have room for 40 nodes in a rack? Betwee TOR switches, storage, and miscellaneous other, it seems unlikely.
To see the difference in practical terms, pick an item that's noticeably warm, but not hot enough to burn you. Hold your finger very close to it. The heat you feel there is radiated. Then touch it, and feel the heat that's conducted through touch. You'll feel much more heat with the latter.
What could Stack Overflow be doing that requires such a dense GPU/CPU configuration? I didn't think a commenting site would required that level of parallel processing.
Also, they do get a lot of comments. You can read about their setup here: http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...
http://nickcraver.com/blog/2016/03/29/stack-overflow-the-har...
Makes sense the server should be really massive.
(Anyone else out there ran Prime 95 and Furmark for fun in the past?)
Sure it does, the energy has to go somewhere. If it's not being stored in some way or emitted as EM then heat is pretty much all that is left. If you measure voltage and current for Vcore going into the CPU then you can easily calculate the amount of heat it's generating since the CPU can't really store any appreciable amount of energy and there's basically nothing else that would allow that energy to leave the CPU.
Overall energy consumption of a computer does not equate to only the TDP of the CPU, itself.
Not really. Modern RAM uses a couple watts. Fans use at typical 2.5w each. Hard Drives only a few watts (SSD variety.) The PSU in servers tends to be fairly efficient, maybe losing 50-ish watts due to heat, and they run more efficiently at lower draws than higher draws. So using a Kill-A-Watt gives you a better idea of how much power the CPU is truly sucking down (especially when a GPU isn't installed.)
This is how I catch LED lighting companies lying about their actual power usage on their lights.
"TDP is not the maximum power that the processor can dissipate." -- Intel
Keep in mind that TDP is only a target number for typical use, not mprime. Not to mention Intel and AMD define it differently, see:
http://www.silentpcreview.com/article169-page3.html
"Intel is listing TDP numbers that are significantly lower than the actual maximum power draw of their CPUs."
That article is THIRTEEN YEARS OLD!
I mean, using that logic the max power consumption for the same six core processor Atwood tested would be around 280 watts. A number only 40 watts higher than his for heavy load. Too bad I used "false information" to reach such a usable number.
Now if the actual power consumption was double that, I'd definitely be berating Intel. But alas, no.
I get your point, but nothing wrong in drawing attention to the specs being incomplete to the point of being useless.
Have fun getting AMD and Intel to agree on something.
> I get your point, but nothing wrong in drawing attention to the specs being incomplete to the point of being useless.
Vague or averaged is not useless.
http://www.intel.com/content/dam/www/public/us/en/documents/...
Intel confusingly tells us "TDP: Thermal solution should be designed to dissipate this target power level. TDP is not the maximum power that the processor can dissipate."
So according to Intel [1] - "The best way to measure a server’s power consumption is the power meter, an inexpensive tool that is plugged into the wall, and then your device, like a server, can be plugged into the power meter. The meter displays the wattage drawn "at the wall" and allows you to analyze the power consumption under a variety of different utilization levels.". Strange.
I was looking up docs on AMD. They seem[2] to kinda get it: "To allow optimal reliability of the AMD Opteron and AMD Athlon 64 processor-based systems, the thermal and cooling solution should dissipate heat from a processor operating at its maximum thermal power.". I could find some old docs[3] that did give the maximum power, but can't seem to find any on their recent CPUs.
[1] http://www.intel.com/content/dam/doc/white-paper/resources-x...
[2] ftp://ftp.sgi.com/public/Technical%20Support/Pdf%20files/AMD/26633_5649.pdf
[3] http://hackipedia.org/Platform/x86/AMD/AMD%20Thermal,%20Mech...
4.1 T_CASE and DTS-Based Thermal Specification Implementation
Thermal solutions should be sized such that the processor complies to the T_CASE
thermal profile all the way up to TDP, because, when all cores are active, a thermal
solution sized as such will have the capacity to meet the DTS thermal profile, by
design. When all cores are not active or when Intel Turbo Boost Technology is active,
attempting to comply with the DTS thermal profile may drive system fans to speeds
higher than the fan speed required to comply with the T_CASE thermal profile at TDP.
In cases where thermal solutions are undersized, and the processor does not comply
with the T_CASE thermal profile at TDP, compliance can occur when the processor power
is kept lower than TDP, AND the actual T_CASE is below the T_CASE thermal profile at that
lower power.
In most situations, implementation of DTS thermal profile can reduce average fan
power and improve acoustics, as compared to T_CONTROL -based fan speed control. When
DTS < T_CONTROL , the processor is compliant, and T_CASE and DTS thermal profiles can
be ignored.
5.3.1 Intel ® Turbo Boost Technology
Intel ® Turbo Boost Technology is a feature available on certain Intel ® Xeon ®
processor E5-1600 and E5-2600 v3 product families SKUs that opportunistically, and
automatically allows the processor to run faster than the marked frequency if the part
is operating below certain power and temperature limits. With Turbo Boost enabled,
the instantaneous processor power can exceed TDP for short durations resulting in
increased performance.
(http://www.intel.com/content/dam/www/public/us/en/documents/guides/xeon-e5-v3-thermal-guide.pdf)
Intel literally spelled that out in this PDF which they linked from their TDP definition on every CPU specification page.
Now, in terms of power consumption. You must consider the entire system as the CPU is going to manage itself to fit its environment given the TCC. That is why Intel suggests you measure overall consumption for the server using a power meter as each implementation can yield different results. Therefore, what Atwood is doing is actually Intel's recommendation for considering actual power consumption. He, as I originally stated who knows how many replies ago, is making the mistake of directly equating power consumption to heat dissipation. When all you can really be sure of is that if a CPU consumes 1 watt of power it can dissipate up to the same in heat. But, as thermal dynamics will tell you, it will always be a bit less on the output as nothing is 100% efficient.
Therefore, all Atwood's test proves is an Intel® Xeon® Processor E5-1650 v3 has the potential to dissipate up to 250 watts of heat while in Turbo, given its recorded power consumption, in that specific computer configuration while running mprime. Nothing more, nothing less.
That is my point.
This is basic PSU and power supply principle.
The two CPUs were in identical test rigs. There is a 80W difference changing only the CPU. While you can expect some of this is lost in the PSU, simple fact is more power is ending up inside the computer and it has nowhere to go but out as heat. The most reasonable explanation given otherwise identical builds would be to expect this difference to be due to the changed component.
So if we assume that Intel's 4-core is actually 140W TDP, then there's no way this 6-core can also be 140W.
Yes, this isn't an exact, scientific test, but it's certainly reliable enough to say "this 6-core processor is emitting more heat than the 4-core, although they are rated identically" which it seems to me was the only point he was trying to make given the context which was "two more cores, slightly lower clock speed, that might be an okay tradeoff - OH WAIT, MORE HEAT".
Why? How can you say that under typical usage the thermal dissipation for both chips isn't the same? Atwood's numbers measure overall power consumption while idle and under heavy load with mprime, neither of which is what TDP seeks to measure.
TDP is like fuel economy for cars. You don't claim it's a lie when you go one hundred and fifty miles an hour for 20 miles and burn two gallons of gas. You simply realize those highway numbers are meant for more of a sixty mile per hour journey over the same distance.
Yes, TDP is an inexactly defined and meaningless term and while to the layman it would generally be understood to have some relation to the heat emitted by the CPU during normal operation, it's possible that both chips in fact only generate 1W of heat and were given a 140W TDP because Intel had a pre-existing cooling solution and a warehouse full of spare parts. Yes, you are correct on the semantics. Power consumption has no relation to TDP because heat generated has no defined relation to TDP.
However, given any meaningfully bounded definition of TDP, the case is still made that the TDP of these chips should be dissimilar. The later numbers show that under full load the power draw at the wall increases by 20W/core used. Unless your "typical load" used for determining TDP does not actually make use of the full number of cores (which I would think could be fine for a desktop processor, certainly not a server) then it's clear that the heat generated by these two processors should be dissimilar under any load.
If the 4 core actually needs to dissipate 140W under a typical use case, then the 6 core should absolutely need to dissipate more unless the "typical use case" is uselessly applied.
If we want to talk cars... Let's say I sell a base model with a top speed of 80mph, and a sport model with a top speed of 120mph. But I tell you you only need to put 80mph rated tires on the sport model because that's as fast as a typical person drives. Would you really claim that the sport model's tires are correctly rated? Would you not be surprised when you drove 100mph in the sport model and the tires exploded? Why on Earth would anyone even buy the sport model if it's crippled to nearly the same performance as the base model?
> Unless your "typical load" used for determining TDP does not actually make use of the full number of cores
> If the 4 core actually needs to dissipate 140W under a typical use case, then the 6 core should absolutely need to dissipate more unless the "typical use case" is uselessly applied.
Taken from Intel's own specs for the E5-1650:
"Thermal Design Power (TDP) represents the average power, in watts, the processor dissipates when operating at Base Frequency with all cores active under an Intel-defined, high-complexity workload. Refer to Datasheet for thermal solution requirements."
Immediately notice "Base Frequency", not Max Turbo, "all cores active", and "Intel-defined, high-complexity workload." Until you can perform the same test on both chips, you cannot assume that "these two processors should be dissimilar under any load."
In regard to your car analogy, if the sport version had a management interface to limit speed due to the rating on the tires, then it would be like an Intel CPU. Read this:
http://www.intel.com/content/dam/www/public/us/en/documents/...
"Thermal Design Power (TDP) should be used for processor thermal solution design targets. The TDP is not the maximum power that the processor can dissipate."
"Analysis indicates that real applications are unlikely to cause the processor to consume maximum power dissipation for sustained periods of time. Intel recommends that complete thermal solution designs target the Thermal Design Power (TDP) indicated in Table 26 instead of the maximum processor power consumption. The Thermal Monitor feature is intended to help protect the processor in the unlikely event that an application exceeds the TDP recommendation for a sustained period of time."
Therefore, are Atwood's test results really a shock? At idle both chips are within 15 watts of each other, yet under sustained load (which is not what TDP seeks to measure) they both violate their advertised TDP.
Again, TDP is NOT peak thermal dissipation.
No. Different manufacturers use different definitions of what a typical load means.
> If not, how would you attempt to test it other than placing the processor under heavy load?
Well given that Intel has stated, "The TDP is not the maximum power that the processor can dissipate." Then testing under heavy load is most certainly not correct.
They're very effective, but must be used with extreme care because aluminum is highly soluble in liquid gallium. Aluminum is the most common material for heatsinks, and it will literally dissolve if the gallium touches it. Gallium is also electrically conductive, so if you accidentally dripped any into your processor socket I assume you're going to have a problem.
If you're careful with it and make sure to get a cooler with a copper contact area, it'll cool more effectively than traditional thermal pastes. I thought about trying it, but it seemed like more hassle than it was worth.
Aluminium is the most common material for heatsink fins. The cold plate is almost always (nickel-plated) copper.
https://www.youtube.com/watch?v=4HKpMYJ-6go
Did he apply it directly to the top of the chip die or to one of those aluminum package covers? Did he use a copper heat-sink? Is somebody's leg being pulled?
The other side (heatsink with fan) probably had to have been full copper (not the pipe/aluminum)
Either way, it doesn't seem worth the effort for marginal gains.
In the old days, I used to tinker with my machines, improve airflow, better CPU cooler, liquid cooling etc. Now I just want stuff to work, so I buy a laptop.