160 Mac Minis, One Rack
hackaday.com
hackaday.com
simbimbo says: December 9, 2012 at 11:03 am Thanks for the great write up Hack A Day. I would like to answer some of the questions posted. @Geebles these machines all run SSD’s and I ordered them with AppleCare, so I hope to never have to change a drive ;-)
As for the reason I built this.. Well, I guess I’m just like a challenge ;-), but seriously, the company I work for has a need to have large numbers of machines to build and test the software we make.
There were plenty of discussions of Virtual environments and other “Bare Motherboard”/Google Datacenter-type solutions, but the fact is, the Apple EULA requires that Mac OS X run on Apple Hardware, since we are a software company we adhere to these rules without exception. These Mac Machines all run OS X in a NetBooted environment. We require Mac OS X because the products we make support Windows, Linux and Mac so we have data centers with thousands of machines configured with all 3 OS’s running constant build and test operations 24 hours a day 365 days a year.
As for device failure, we treat these machines like pixels in a very large display, if a few fail, it’s ok, the management software disables them until we can switch them out. This approach allows us to continue our operations regardless of machine failures.
@bitbass I tried the vertical approach, but manufacturing the required plenum to keep the air clean to the rear machines cost too much for this project, but it’s not off the table for the next rack
@Kris Lee When I open the door I can literally watch the machine temps go up, but I can keep it open for 15-20 minutes before the core temps reach 180F
@Adam Ahhh.. Nope, you can’t have my job ;-)
e: Ah, according to [this article](http://www.wired.com/wiredenterprise/2012/05/mozillas-new-da...) they have 500 Minis in their data center.
lol
It might have been contributing if you included the codinghorror link up front.
In response to said link, I will observe that it was written 1.5 years ago, and SSDs have been progressing very rapidly in both performance and reliability.
Looks amazing! Some real professionalism there! However replacing a hard drive is going to be a pain! Wonder if there is a way to modify the mac mini to allow a more accessible hard drive access?
So I read his reply as separate statements: a) They're all applecare covered, and non-vital so he won't need to muck around with maintaining the individual mini's, and b) They're running SSD's not HDD's.
You could have avoided the hail of downvotes if you had of included your conclusion that you read the applecare part but still understood it as he was saying the SSD's were more reliable. I don't think I've ever seen a comment of just 'lol' escape downvotes on HN, it's pretty clearly not good HN etiquette.
Guest: Leopard Server
Host: ESX 5.1
Hardware: Mac Pro
Mac Pro is supported on the VMware HCL. Leopard Server is legally virtualizable on Mac hardware. VMware supports OSX Leopard as a guest on ESX 5.1
Mac Pro towers are going to be less dense, but given his cooling situation, lower density is probably a win. What datacenter wants 8 kW of laptop CPUs stuffed into a rack? Virtualization would also overcome the lack of redundant PSUs.
Here is a post that Jay Parikh (VP of Infrastructure) made about it. http://tinyurl.com/cnvss4v
Our density isn't as high (we have 64 minis) because of cooling and cabling that we designed according to our datacenter cooling standards.
@jurre - If you want to chat about our design, message me and I can put you in touch with our hardware designer.
I don't work on the team anymore, but I can probably start off a thread with the right people involved from Facebook's side.
edit: I guess lumped into this is the small market that seems to exist for colocated Mac Minis. Is there something about them that is better than renting commodity x64 hardware?
* Launchd is a massive improvement over the equivalent mess on Linux. This can't be understated if you are managing your own hardware. * You can develop on the same machine you are deploying to. * You have exactly the same toolchain as on Linux. * Lots of remote monitoring options that are unique to OSX e.g. OSX Server * The OS is stable and upgrades are safe enough to enable auto update. I could never do that on CentOS.
But really it comes down to hardware and resale value for me. 2 Mac Minis in 1RU is great value.
OS X doesn't have the same toolchain as Linux - notably, Apple doesn't ship anything licensed under GPLv3. This leads to OS X versions of common GNU tools being extremely outdated.
I do agree with the point about developing and deploying in the same environment. I do both in Linux for that exact reason; far fewer surprises when your deploy environment matches what you've already debugged.
Nobody uses what Apple ships by default. Homebrew or MacPorts and I have exactly the same setup as I have on my Linux server. Plus lots of amazing GUI wrappers which can simplify setup and ongoing maintenance.
My biggest problems with the above:
1. Neither Homebrew nor MacPorts (nor fink, for that matter) has a complete set of relevant packages
2. Compilation from source is ridiculously slow.
3. Certain utilities are best installed from .dmg files instead, and it's a pain to remember which was installed which way.
Man, if I subscribed to that belief, I might as well just use Gentoo!
Especially for socket/mach service activation. This feature is integral to both iOS and OS X, and is dead simple to configure.
1) It unifies several independent systems that shared a similar purpose:
- init
- crond
- atd
- inetd
- xinetd
2) It includes user-based daemon management.
- More structured than doing it in your shell's RC file.
3) Instead of writing a shell script, you set a series of properties in a plist (XML-based).
- It's parseable.
- Daemons can be interacted with programmatically using built-in tools like defaults, PlistBuddy, and launchctl.
4) It handles a lot of special dependencies:
- Network availability
- Disk or server availability
- Filesystem availability
- User Logins
- Kernel Extensions
5) It has built-in, live, event-based triggers with almost no overhead. It can watch for:
- Changes to a file or folder
- An item to appear in an empty directory (for queueing)
- A file system to mount
- A Mach message
- A connection to a stream socket
- Traffic to a datagram port
6) It manages respawn behavior. For example:
- Run when loaded and never quit.
- Run purely on demand
- Run once when loaded an on demand thereafter
- Run on demand based on a clean exit
7) Its strict conventions prevent processes from being disassociated from their parents.
- Easier to trace who did what (nice for security and troubleshooting)
- Easier for the OS to garbage collect.
- Centralizes a lot of daemon management
8) It has a defined separation of "execution contexts" based on:
- If it does something for the currently logged-in user or all users.
- If it will be used by one app or by multiple apps.
- If it needs to display a user interface or launch a GUI app.
For more info, checkout: http://developer.apple.com/technotes/tn2005/tn2083.html
EDIT: Formatting.
I don't quite understand the other points; you can certainly develop on Linux and there are lots of mature monitoring options.
The trick with auto-updates is, you're getting everything from the package manager. This forces you to keep everything up to date, not just some notion of the 'core system', which is what you're getting from Mac update. Of course, you have to have some discipline and stage major updates in a testing area before you actually update production servers, but I think you'll find key packages like OpenSSL should probably be kept up-to-date.
Personally, I maintain a system which consists of about 30 CentOS servers; not large by any means. But things like upgrades and monitoring are a non-issue; we test updates before we apply them, and we use Nagios for monitoring.
As someone downthread pointed out, you can get a SuperMicro 1U server with a hell of a lot more horsepower than a Mac Mini for an equivalent price; the motivation for Mac servers seems to be almost exclusively to test and build Mac-specific software. I'd be curious to know what your specific application is that motivates this?
Yes, if you make software that runs on OS X (or iOS and you want to test on iOS Simulators) you need OS X machines for your build and test process. You need lots of machines so you and your fellow developers can run speed things up and run tests in parallel.
Also they rack mount very easily: http://www.sonnettech.com/product/rackmacmini.html
The worst part is that some failures were not a 'stop dead failure' where it was easy to detect and replace. Some would just freeze intermittently, some just got really slow.
One could tell that there is a practical difference between server grade and consumer grade components. In retrospect time spent debugging and messing with this was probably not worth the gain in density and the cost.
I've only got a sample size of two recently purchased (2012) Mini's, and they've both had memory issues.
Oh and we get 640 cores in 20U (8x4 core xeon machines each 1u) and that leaves enough room for a 32Tb SAN, FC switches and a pair of redundant LAN switches.
REgarding splitting the power using the hack described, 160 melted minis and a halon cloud coming up.
Looks pretty though.
You should have a talk with your power cord provider. You should be using cables that can handle at least 4 amps in anything with a 110v plug on it. I don't think you can buy one smaller than 18ga and those are good for 10 amps. Remember, you have to handle enough current to blow the breaker if something goes wrong (unless you are British and have your own fuse in the plug).
Is splitting the power really so dangerous?
[1] http://support.apple.com/kb/HT3513 [2] http://support.apple.com/kb/HT3468
Most rackspace rented (in UK DC's at least) tend to be a maximum of 16A (at 240V) per 42U cabinet, so just under 4kW. By my estimation those Mac Minis will be drawing ~13kW at peak.
The problem is you often end up paying for the rack space that would be allocated for the total power you're using, without actually getting the rack space!
In exchange, the DC's own electrician turns up during your install and makes sure things are working correctly and safely, you might end up with different sockets if required, 3-phase power, etc.
Even if the resistivity values of the copper and solder are similar (I'm pretty sure they're at least in the same order of magnitude), you'll still end up with the same amount of current running through the copper wire and the solder.
In any event, 220v is no big deal. It's household current almost everywhere except the US. (A few countries use 110V, Australia is supposedly 240v.)
I was under the impression that the issue at hand was that using solder for high-load wires was bad because of solder's extremely low melting point. I meant to point out that even though the solder is just holding the wires together, there is still a current running through it, and if the resistivity of the solder is lower than that of the copper, the chances of the solder melting would be quite a bit higher.
I honestly don't know if that's why the parent parent post mentioned that soldering high load wires are bad. I just extrapolated issues that someone might have when using solder on high load wires, and relayed them in my reply.
And what's the problem with that? The resistance of the solder will be minimal (as you say yourself, pretty much equivalent to copper), so the voltage drop and heat production should both be low? Am I missing something?
I mean, I'm pretty sure there's solder connecting the copper wire to the PCB of any PSU in my house, all running at 230V.
Pure copper has a resistivity of 0.0172µΩ⋅m while 63% tin/37% lead solder has a resistivity of 0.145µΩ⋅m. (http://alasir.com/reference/solder_alloys/) That's almost an order of magnitude difference.
Let's say you had the absurd case of 1/2 copper and 1/2 solder. I = I_cu + I_solder, and V = I_cu * R_cu = I_solder * R_solder => 8x more current going through the copper than the solder.
You would need 8x more solder than copper in order to get "more current going through the solder than the actual wire."
You forgot to take into consideration the thermal conductivity of solder. Although there's only 1/8th the current going through the solder than that of the wire, solder has a much much lower thermal conductivity than copper. After a bit of googling, copper's thermal conductivity is 401W/(mK) and bismuth solder is 19W/(mK). Although there's less current going through that solder, it's so much more thermally conductive it'll probably see its temperature increasing faster than that of the copper.
That certainly doesn't help when solder's melting point tends to be around 140C and copper's is 1000C. Also, while you may have said a 50/50 mix of solder and copper is extreme, if the OP was an awful at soldering and was using something like 16 gauge wire, it's not so ridiculous that there may be a 50/50 mix of solder to wire, hell that might even be a bit low.
The numbers you yourself reported show that copper is much more thermally conductive than solder, and not the other way around.
Thermal conductivity describes how quickly the heat conducts through the wire. Since the wire is uniformly heated, this is a minor detail (assuming that the wire's thermal conductivity is considerably higher than the electrical insulation around the wire). Instead, you need to look for the heat transfer coefficient of insulated wire.
The melting point of Sn63Pb37 is 183C, not 140C. 183 is not "around 140."
If the wire were hot enough to melt the solder in a copper+solder combination then it would be well more than hot enough to melt the plastic insulation around just the copper wire itself, which is typically rated for only 90C.
Household wiring must always use screw fittings, wire nuts, etc. Solder is not "generally approved" on dc, either, unless inside an approved housing, and then only when done in certain ways.* Household wiring is all the construction codes usually bother to cover, and they apply to 120/240Vac up to the switch, socket or outlet. Lamps, stereos, etc. are covered by code in some areas, like LA county, but by "Underwriter Labs Approval" in most others, in the US. A "UL" label (o/e) on the power cord is required for sale in most larger US communities.
120VAC can be attached by solder to an approved type of glass or phenolic circuit board inside the correct kind of enclosure. UL generally will not approve its use just about anywhere else for power-line connections.
This is something to keep in mind when doing any DIY project. Get a copy of the UL construction-code book. You'll find that all connections should be inside a non-meltable (fire-retardant) enclosure, via screw terminals, wire nuts, or compression fittings of some kind. Heat and humidity can quickly cause electrolytic corrosion on any soldered joint, eventually causing enough resistance for it to get hot and melt down.
Solder is great for some low-voltage dc purposes. When a connection might get subjected to heat (as in a house fire, resistive connection, etc.) the mischief that running solder can cause is just too great for it to be a good idea, so codes do not approve it in most areas. [Sorta like teflon tape in high-pressure gas lines. :-)] Over time, electrolysis slowly destroys soldered copper connections that aren't hermetically sealed, too. Some fungi accelerate that.
This is for reasons of mechanical strength. Naive soldered connections can be pulled apart with your bare hands, and will not stand up to being snagged, tripped over, used as an acrobatic toy by a toddler, etc. This is especially true for the single-strand cable used in building wiring, which can apply a lot of leverage to a soldered joint.
Electrolysis and fungi? No, solder is chemically very stable. It's used to great effect to join copper water pipes, where the large surface area and built-in strain reliefs allow solder to be strong enough.
It draws precious little electricity, in other words.
A test failure will probably bring up an engineer that will track down the issue, and a re-test will inevitably occur. The faulty machine will eventually (hopefully) get labeled flaky and will get repaired.
Of course, nobody may care and just use a double-test to verify that an executable is good.
Depending on how valuable the engineers' time is. I have seen this played out like this: hardware gets blamed last after hours and days of testing have been wasted. So tests are run and re-run, blame goes all around until finally after hours and hours of testing it is determined that maybe it is hardware after all.
In the end an engineers' time is worth a lot more than savings obtained by running flaky but cheaper hardware.
It would be better to have the fans on the back and suck air through the rack rather.
That said, DC floor space is cheap compared to power and cooling. I'm surprised they didn't lower the density so as not to have a massive fire risk.
The fans are large so they move a high volume of air at a low speed, the door doesn't move is left unlatched.
Also, can you please explain your "Massive Fire risk" comment? All of the hardware installed in this rack is UL certified and all of the machines will simply shut down if they get too hot.
This would solve his thermal dissipation problem and probably be easier when compared to getting custom hardware.
Plus if you want to upgrade them then you can put them on eBay and get 75% of the original cost back. Try doing that with a server.
The closest thing to it (Mac mini: 4 core i7 / single SSD / 16GB ram) costs $1499 from Apple.
Are you proposing that I can't get a dual i7 @ 2.6GHz, dual SSD, 16GB ram from Supermicro for less than $3,000?
I have a mini on my desk, but even as an Apple fan, I think you're off the mark here.
Then remind me how much it is going to sell in a year when I decide to upgrade. I guarantee that the Mac Minis would sell in a day on eBay.
However, you're not going to get dual-NICs, ECC memory, or multiple "spindles" in a Mini formfactor.
But for use cases where you just have dumb nodes they are perfect.
Any beowulf setup would be cheaper. But, as mentioned, the guy had special software constraints.
http://www.supermicro.nl/products/nfo/FatTwin.cfm
http://www.supermicro.nl/products/nfo/2UTwin2.cfm
http://www.supermicro.nl/products/nfo/MicroCloud.cfm
Those would all be cheaper than the Mac Mini
I love the mini (I have one under my desk!) but suggesting apple consumer gear (including the apple tax) would be a cheaper alternative to generic x86 consumer-gear is a little nuts.
I just summed up the component prices for similar servers that we have built - that's the ballpark if you are willing to assemble it yourself (takes around 45mins when you're not doing it the first time).
For a pre-assembled box google found me this (first hit): http://www.nixsys.com/products/intel-rackmount-servers/intel...
That comes out at $1735 for a 6-core, 12G Ram, 128G SSD or $1385 if you use the 4-core equivalent to the mini.
Rather expensive but in the ballpark and already cheaper than the mini (if you buy RAM and disks separately you can probably shave another $300 off).
At some point if the need arises, I'd love to look into getting a thunderbolt nas for the db storage. That would be fun.
That and the lack of ecc memory is what currently holds me away from buying mac minis for hosting.