Six-Year homelab history in pictures | Hacker News Reader

Six-Year homelab history in pictures

blog.networkprofile.org

119 pointsdevy61 comments

This is very awesome. For those interested, the Reddit.com/r/homelab section is super great for this kind of stuff.

I just got a Dell R720 off Craigslist to do an experiment with the Firecracker MicroVM of all things. It's got 48gb of ram and dual hex-core Xeons. It's a lot of fun to play with this kind of gear.

Ikea sells a 'rack' (not the Lackrack, https://wiki.eth0.nl/index.php/LackRack) that is really great for holding gear: https://www.ikea.com/us/en/catalog/products/S39031411/

Some pix of my recent setup: https://imgur.com/a/xXc6720

Back in ~2014 I had a similar tiny setup in my basement with a UPS and an older (and much louder) Poweredge 2950: http://i.imgur.com/zxYxuLq.jpg

The Dell C62x0 series high density compute systems are pretty cool too. Up to 4 nodes per chassis, dual cpu 48 gb per node.

Biggest issue is fan noise. You have to tuck them away somewhere.

e.g. https://www.ebay.com/itm/Dell-PowerEdge-C6220-GEN-II-4-x-Nod...

Since I got fiber gbit up and down I try to relocate as much as possible of my homelab to real data centers (Hetzner) since I can't stand the noise. (I have 10ms ping to my server)

Then I use wireguard to connect it to my home network and make it act like a server in the home network. (All the traffic even gets directed through home)

I still have the main storage server locally though because otherwise I'd not be able to even watch movies if my internet went down.

My dream would be an all SSD storage server being near silet.

What fiber did you get?

I live in Switzerland so I have Sunrise.

Awesome - thx for sharing.

I recently set up Pfsense on an old box I had lying around. The practical uses i’ve found are: setting up the PfsenseNG DNS adblocker and setting up an OpenVPN server so I can VPN into my LAN away from home instead of port forwarding. I also just set up Dynamic DNS so that I can do this with a domain.

Can anyone share other use cases they’ve found to be practical with their homelab setup?

To expand on the VPN benefits you already have in place... grab a vnc app for Android and VPN in then vnc to a machine from your phone. All of the sudden you are sitting at an Linux desktop on your phone. With DLNA casting you can literally now use your phone as a full blown computer casting to a monitor (assuming you have some sort of peripherals). That was always my holy grail.

- Fine-grained QoS controls via PfSense

- Guest Wi-Fi VLAN with UniFi + PfSense

- NAS with automatic backups of client machines via Syncthing

- Automatic file history of backups with ZFS Snapshots via FreeNAS

- Automatic mirroring of backups to the cloud with Duplicati + Sia

- Home media server via Plex

- Home automation / web dashboard via Home Assistant

- Minecraft server with MineOS in a VM

Those are all things I either have working, or am currently working on setting up on my own network.

Can you expand on the QoS setup you’re using? I’ve recently read about the fq_codel addition to Pfsense and wondering if anyone here has been using that.

I’m also looking for more performance related tunes as well. To that end, I’ve configured Cloudflares DNS but not sure what else yields results. It sounds like squid caching is mostly useless due to most traffic being HTTPS.

It's actually a pretty simple queue-based setup right now. My primary intent was to ensure that bulk downloads and uploads don't interfere with my latency while gaming.

I did look into fq_codel but I don't believe that was a practical option on PfSense back when I was setting everything up. I might have to revisit that now that it has native support in PfSense 2.4.4.

fq_codel is working pretty well for me to keep my ping consistent with big downloads going. There are some straightforward tutorials out there now too, since pfSense has native support.

I have a virtual FreeBSD box running dnscrypt-proxy. It's configured with CloudFlare for DNS over HTTPS.

"I then got hold of a 3.2TB Micron 9100 PRO NVMe SSD"

How does one "get hold" of a $3k SSD? This is awesome though, you clearly put a lot of work into this.

What do you store on the servers and use them for?

Not my lab but it varies from homelab to homelab. A lot of it is using it as a safe place to understand/experiment with new tech without breaking anything in a production/work environment.

Some people also use them to run Plex/Pihole/Pfsense and the like as well.

I think this is a pretty good summation. I'm fairly active at the r/homelab subreddit and this is pretty on point. Usually there is some combination of upgraded home media/routing and then on top of that a learning and experimenting section based on your specific goals.

For others though it really is a hobby in the sense of building to build.

linux isos

I use mine simply as a hobby. For a while, I tried to justify it to myself that it was "for work". But it wasn't.

I play model datacenter the way my grandfather played model railroad. Good news is that I spend way less than he did. The future is awesome.

Certification prep mostly, some concepts are best not tested out at work.

With that amount of storage, the most likely reason for all the gear is hosting media.

These days, I mainly use three old i5 boxes with 8GB RAM and SSD RAID10. Two with 4x128GB and one with 6x256GB. They're all VirtualBox hosts, and run various pfSense VPN-gateway VMs, Whonix VMs, and other workspace VMs.

I also have some old rack-mount servers. But their fan-control systems are all Windows-based, and they're impossibly loud with Linux. I keep meaning to do some crude electrical hack -- basically a manual speed control -- but just haven't gotten around to it. And there's also the fact that I don't need that much muscle locally. Because I like to keep my network activity minimal.

Everything is on a 0.25kWh UPS (3kW maximum load). With a kill switch on my desk. The UPS and a pfSense perimeter router/firewall (and a rack with the old servers) live in a well-ventilated closet.

Nice write up and photos. I used to maintain a lab like this only my cable management was nowhere near as nice as his. I've replaced this all with a combination of AWS and an Antsle. I miss the hands-on hardware time but don't miss the fan noise.

Looks like a damn expensive hobby, one which I have a certain affinity for.

Usually people buy old servers instead of building supermicro. You can pick up an R610/R710 for $2-300 and even the newer R620/R720 for $3-400.

As long as you doing mind the noise + power they're not a bad deal.

As others have said, it can be done cheaply. The electricity costs more than the hardware at least for most people's setups that I've seen. (Exception is people using stuff like RPi boards from solar panels; that's almost a sub-hobby in itself.)

I run mine in a basement in place of a small utility heater. If I'm going to burn the watts anyway, might as well get some processing cycles out of it.

Be patient and wait for deals you can snipe. You can get a server that was considered high-end circa 2010 for $50.

Some high-end Xeon chips are dirt cheap, like $50 each or less, as parts. Certain models were made by the millions and are still good, but got tossed as more energy efficient versions came to market. It's just not economical to use them at scale.

I used to run a full size rack with servers and gear but ended up downsizing to three NUCs with 32GB in each, a simple managed Ubiquiti switch, and an 8 bay Synology like the one in the article. Doesn't have the "wow factor" of the big rack of equipment, but it's a lot easier to work on, quieter, and cheaper on the electric bill.

Worried to see the rackmount hardware close to the carpeted floor with a wall directly behind them. Potential for dust and airflow issues.

I've had bits and pieces of a home setup over the years (currently just an HP microserver with disks and VMs) but lots of cost and noise for something I probably won't use a lot of.

Cool. This puts my homelab to shame. I've basically got an old i5 server running Linux (non SSD), an unstable HP DL160 (never buy HP, kids!) and a Raspberry Pi for home automation. I've also got lots of additional hardware that I rarely power up such as Sun Blade 2000's, SGI's etc.

I've been trying to get rid of my PowerPC and SPARC machines forever... it is a very slow going business.

The casual handgun on the desk in pictures is quite hilarious.

You have to protect your data from both remote and local attacks.

My problem with homelabs (mine and others) is the power management. Finding hardware that triangulates to energy efficient/good proxy for production/cost effective is really hard to do.

Noise is the other issue. I found a nice fanless switch, but my server was noisy for years until I got serious about fans and switched everything to 4-pin Noctuas with PWM set at 50%.

I made my server near silent (it's basically just a desktop computer) but now the biggest noise comes from the disks. I can't stand the noise. Not sure how to make disks quiet though.

Get some SSDs

Play with the acoustic settings or switch to NAS-grade drives (WD Red)

I just ignore it. I have about 10 servers in my basement. Total electric bill is like $160/month.

I've got a T710 pulling about 200W continuous and it contributes (.28 * .2 * 24 * 30) = $40/mo to our electricity bill in California.

A Hetzner dedi is more cost effective for some workloads.

We had a small cluster in the basement, with a server for all the backups, a firewall etc. initially for our software development and later mainly for email.

I just decommissioned it all (still have a few network hard drives for local backup of laptops etc). The power it was drawing was about 300-400 watt. That is 7.2-9.6 kWh/day and 2600-3500 kWh/year. (At about $0.1/kWh.)

That is about equivalent of powering a Tesla Model 3 (160 Wh/km) 16000-22000 km, which is about what I drive per year.

Our garage has a south east facing roof which with solar panels could deliver about 3000 kWh/year. That installation isn’t terribly efficent due to scale, and would cost me about $15000 to install.

Instead we decided to move the email to a Swiss email provider (Migadu) that cost about $4/month. We may install solar panels later amyway, but moving the servers into the cloud cost $48/year and gave us enough equivalent electricity to drive the whole year.

"After messing around with the ERL for months and going through flash drive after flash drive (What they boot from) I decided it was time to move on."

Hmm, if he is talking about the EdgeRouter Lite, I have one, going strong since ~2014, didn't have any clue it has problems with flash drives, and there certainly isn't any need to open the case and mess with internal components. It's a good little router and I have few complaints with my Ubiquiti gear. I have not placed mine in a rack without adequate ventilation, however.

I've heard the same complaint about the USB flash drives failing. I run an ER-8 but have had no issues. I think it may have been a bad batch or something.

Some people have also had issues after installing OpenBSD on them without limiting writes.

Anyone who is thinking of doing this should consider how loud and annoying 1RU sized servers and switches are. They're not designed to be used in the same room as people for any extended amount of time. If you have a whole room in your house that you can dedicate to this, or a corner of a garage, sure. But don't buy a used Dell R610 1U server off eBay and expect to put it comfortably in your home office.

Agreed. Workstation towers like the Z420 are probably a lot quieter for small homelabs. I have a TS140 and Microserver, while the old Microserver is nearly silent, the TS140 can be irritating if seeking a dead silent room.

I, on the other hand, use a very old pink netbook I got from a garage sale to host my syncthing backups and as a CalDAV server.

Hey, it even comes with a USP!

I love this idea. You can get decent laptops (i7, 8gb rm) with broken displays for cheap on EBay/Craigslist, and install Linux. Best way to set up a homelab on the cheap.

This was actually one of the most cost efficient ways to do scaled low power datacenters for the longest time.

Set the fans to max and do direct DC off of a DC UPS.

I used to run a super loud HP server I got off craigslist for something like 70 bucks. Thing was a beast. Since have switched to a dual Xeon storinator with 30 8tb hdds. My next project is getting another storinator and finishing wiring the house for 10ge

+1 for using old equipment for home labs.

There’s lots of perfectly good hardware out there for cheap that may be beyond it’s useful life for top end production workloads but is still great for other projects or just tinkering.

That's an amazing setup.

My little TS140 with it's Xeon running Proxmox does just about all I need. My Synology does the backups and my EdgeRouter-X does the routing to round things out.

I've always wanted to do something like this. Where would someone go about learning how? Would you need a background or are there other resources you could use to learn?

Well you probably need to have some basic ideas of computers. Then the rest is just lots of googling and trying out etc.

You can also just use VMs for this.

reddit.com/r/homelab has a decent wiki for the hardware side. For the software side, it's mostly DevOps/sysadmin work. DigitalOcean has extensive tutorials for that kind of stuff.

I just have a Skylake Xeon E3 quad core from 2016 as my main sever and an i3-6100 with 8GB of RAM and Intel NICs as my router, but I want more goodies :D

This (wo|)man is living the dream.

“Person” is way more inclusive and way less clunky.

for someone who has zero knowledge of networking, looking at those pics - WOOOOW!!

If you are actually interested in learning more about networking, i cannot recommend starting your own homelab enough.

networking seems to be a hit or miss for people, some people find it very logical and "easy" while others seem convinced it's black magic.

networking basics is rather easy, the problem is complexity that gets added when you scale. (although i doubt you would reach that point in a homelab though).

I have slightly more than zero knowledge of networking and would like to remedy this.

You seem convinced it’s not black magic, so recommend a book/mooc/resource that’ll prove it. :-P

This will explain the basics of LANs in a fun, but very thorough way: https://www.amazon.com/All-New-Switch-Book-Switching-Technol...

And this one will fill out your knowledge on the routing side, although it is rather dry: https://www.amazon.com/Network-Routing-Algorithms-Architectu...

Reply on news.ycombinator.com