Ask HN: Buy a used 1U as a compiler server | Hacker News Reader

Ask HN: Buy a used 1U as a compiler server

8 pointstubby1234526 comments

Lately (last ~6 months) I've transitioned to working on large C++ projects (e.g., LLVM). Naturally compile times are long and a serious barrier to learning/iteration. At work I have a beefy workstation (32 core threadripper, 128GB RAM, 2TB ssd) so I don't feel the pain as much. I'd like to get something comparable at home but I don't want to spend 5k or whatever. Looking around on ebay you find lots of decommissioned 1U units that are pretty cheap, pretty old, but multicore. Would it be a bad idea to get one of these if I'm going to use it just to offload compiling (either through vscode remote or by some other means)?

View on HN

I'm running a decommissioned rack computer in my basement. It works good with some caveats and recommendations:

* Buy a 2U server, not a 1U server. The 1U server fans are VERY LOUD and very annoying. The 2U server fans are still pretty loud, but more tolerable. If you are in the same room, even the 2U servers can be annoying.

* Check the CPU benchmarks (e.g. https://www.cpubenchmark.net/). Older systems with older CPUs might not be as fast as you expect.

* Check how much RAM the system supports and get as much RAM as possible when you buy the system. I had problems trying to add RAM sticks that had the same specs but the BIOS would not accept them because they were not identical (it apparently matched manufacturer IDs as well as timing).

* Efficiency will be less than a new CPU; same or higher wattage with slower performance.

Long story short - they're also SLOW.

Like others have said, the fans of a 1U are going to be LOUD. Consider a 2U.

DDR3 and older DDR4 Xeons are cheap but power hungry and slow. Compare costs with a more modern system, like a decent Ryzen. Saving the money on the older system might not be worth it, when you take power and noise in to account.

I bought a Ryzen 2700X on sale a couple of years ago for large compile jobs. I chose an ASRock motherboard because most if not all support ECC, and I got some ECC memory. It has been absolutely wonderful to have a machine dedicated for CPU intensive tasks.

I agree with the other commenter who recommended a cloud solution. If not though, a 5950x will be way faster than any decommissioned unit in a similar price range. The 5950x is 16 cores around 5GHz and it's on Amazon for $770. 64GB of Unbuffered ECC RAM and an ASRock Mobo and you can do the build for $2k. You can do it cheaper too if you go to 12 cores or 32GB RAM or whatever. If you know how to overclock RAM (or learn how to use Ryzen DRAM Calculator), you can squeeze out another 10-15% performance in compile tasks.

Here is an example build: https://pcpartpicker.com/list/MnFh68

And 4 of these for ECC RAM: https://www.newegg.com/kingston-16gb-288-pin-ddr4-sdram/p/N8...

When faced with the same issue I decided to just setup VPN to the office so I can offload stuff to my office machine when I’m on my laptop at home or traveling. Maybe that’s possible for you too?

This is the best option

I have built a server at home for working, but in the end I just use it as an ssh server, to reverse RDP to work

Option 1: your office has a VPN, ask for access

Option 2: get a cheap 2nd hand server * in UK there are cheap HP ML310, 50£ * Open your home router port * Setup a dynamic DNS

* Setup a VPN or SSH * From the office connect to your VPN/SSH * From home RDP/x2go/SSH to the office computer

Option 3: rent a 2.50£ a month VPS, with a static IP. Setup as above...

I have two servers I use regularly although I have moved them out of my house recently.

I didn't find the fan noise or power usage bad. Startup fan noise on an HPE server is bad but with decent airflow they are pretty quiet really.

Server specs can be different but it might work to your advantage, my HPE servers can use registered ram, are super easy to upgrade and walk all over my older i7 based laptop for compile speeds on big projects. I buy from techbuyer or ebay.

Note that the drive arrays might be SAS rather than SATA, and windows 10 won't install on a fat multi cpu server as the license doesn't permit it, so if you are running windows you will have to install windows server instead. If you need docker like I do, it ended up being better to install debian and use qemu to run windows stuff.

Get dual power supplies. A lot of these servers have lived in data centres and the caps in the power supplies may have silently failed, but you won't know until you try to turn it on. The power supplies are hot swap usually and run to $25 dollars so I keep one on the shelf.

The deskside server I bought is an absolutely kick ass workstation usinga very cheap ati hd6570 and a couple of 2k monitors. Its fast (dual 6 core xeon @3 ghz), super reliable, 96gb of memory. Using a pair of hybrid drives in raid 0 for storage. The other one is a rack server on a big SSD and it is even better but less convenient as a workstation so I use it as a build machine and for hosting VMs.

I would never go back to buying a deskside PC. I am under $1000 invested in those two beasts and to try to get the same performance you are lookung at $5000 and climbing. I recommend old servers unreservedly.

https://www.techbuyer.com/au/

The noise will be simply horrendous. I bought 4 some time ago, and they sounded like a jet taking off...

What type of node did you get if I may ask? I put my hardware in the basement but they are getting a bit old (C6100 machines, which have 4 blades in a 2U form factor).

I have a 32-core 2U system I want to sell. $250 + shipping. Supermicro 2U Server H8DG6-F 2x AMD 6272 2.1ghz 16 Core CPUs, 64GB. It has 12 drive slots but the controller (SAS) only supports 8 drives. You can probably connect the other 4 drives with on-board disk controller ports but I haven't done that. System has 2x power supplies (the quieter ones, though not what I'd call quiet).

I added an expansion card for 2x USB3 ports in addition to the USB2 ports on the motherboard.

I have 8x 400GB Toshiba 15K RPM hard drives for it, in drive sleds. $400 for all 8.

I'll throw in a Happy Hacker mini keyboard or full-sized keyboard.

If interested, my email is in my HN profile. I have the original Supermicro box with foam inserts for shipping. Pics available too - just ask.

As others have said rack servers are really loud. Instead look for a tower server, they are much quieter as they are often used in office environments. Something like a Dell PowerEdge T series. You'll pay more as they are rarer but you'll be happier.

Another option is to build some. Last summer I built a couple of servers for my home lab. I went with a 12 Core Ryzen and 64GB of DDR4 RAM. I spent around a Grand each of them. ASRock Rack makes an AM4 motherboard that has IPMI. So they have remote IP-KVM and power control. It is similar to the Dell iDRAC or HP iLO.

At work we used https://distcc.github.io/ to offload large C compile jobs to our "jet" which is a 4U/8N Supermicro decommissioned machine with 2 x E5-2680v4 CPUs, 128 GB RAM, and 6x 960GB Crucial SSDs per node (224 cores total). It's linked with a Mikrotik CRS-309 10GBe switch to our main gigabit switch. The server is VERY VERY loud and idling at 500 W / compiling at almost 3 kW. The price was about US$10k for all of it. We also use it as NAS / backup for some things since it runs on ZFS raidz2. Very much worth it (using it as a team) but YMMV.

how does this work with ctypes and things like that (or clangd)?

You mean for editing? The part I worked on was not very large, so I had source mounted as NFS share. It worked fine, but I don't use much code intelligence. I think for the larger projects we had a local git server with some hooks to upload and compile. In the local network it works fairly fast.

Keep in mind thst thise things are LOUD and they get LIKE REALLY REALLY HOT.

Cooling is only half a problem in server world because it's assumed that you're going to run servers in a climate-controlled room.

Also, their power draw is not negligible, especially under load.

These are the factors that you should really keep in mind, when evaluating with respect to your specific situation.

For example, elextric power is fairly cheap in the us but quite expensive in Europe. Your neighbours might have something to say if you live in an apartment and have server grade fans running 24/7 (*).

If after all considerations you still want to do that, /r/homedatacenter is full of porn you might enjoy.

(*): I know you're currently not thinking about leaving the server on 24/7... but you'll be tempted :)

Every time I've looked at something like that, it just hasn't been worth the hassle, and hasn't been economically competitive with getting a dedicated server on Hetzner or Hetzner auction. Those boat anchor servers on ebay will be WAY slower than a modern threadripper. Or if you only need a few hours at a time, Hetzner cloud is competitive without having to worry about preemption.

1. Stick with DDR4 systems, DDR3 are much slower. 2. Used to have both 1U and 2U servers at home, as people mentioned they are pretty loud, try to avoid 3. Regular commodity tower case with slots for 120 fans is best in terms of loudness/cooling capabilities 4. Try build something on commodity hardware pieces - would be easier to sell it away once you want to get rid of it.

What about a dedicated Hetzner or OVH server your employer would let you expense the monthly cost for as an educational expense or similar?

Will be noisy. You can pull the board and put it in a desktop case with desktop fans, but unmodified even in a garage the strongly forced airflow in a 1u compute server sounds like an aircraft taking off.

Do NOT follow through with this plan. 1U servers are incredibly loud and you will regret it every day. Plus, like others have said, any old 1U server on eBay is going to be much, much slower than anything modern. You'd be better off building a little silent desktop with an AMD in it.

Are you opposed to the cloud? You could get an even beefier machine on the spot market for under a buck an hour while you’re using it, then turn it down when you’re not.

Do you have a link to this unicorn cloud machine ? I'm very interested for something faster than a bare-metal 32-cores threadripper with 128Gb of ram and a 2TB SSD (NVME I guess) for less than $1 an hour.

An M6a on AWS has 128 GB of RAM and 32 cores and goes for around $0.60 on the spot market. If you need local SSD and you’re willing to go ARM, an m6gd has a similar spot price and nearly 2 TB of NVME local storage. I’m sure other cloud providers have similar offerings.

I agree with the root comment though that you may want to look at building in a cloud provider using spot instances. You’d have to invest in automation to spin up and tear down machines and manage build artifacts, but it might be more cost effective and operationally simpler than investing in your own hardware.

32 cores on AWS is quite different than 32 cores on a thread-ripper no ?

Yeah, it doesn't involve having a noisy new space heater in your home. :)

I haven't benchmarked these instance types, but in the past on the common instance types, t2 from memory, an AWS v-core was similar to core on a very low end cpu.

Reply on news.ycombinator.com