I personally have 4 boxes stacked in a corner that are connected to a "rack", which is just two rack side panels bolted to the back of a workbench that I screw components into, connected to NAS and multiple Raspberry Pi's on a shelf and I really like the mess and have learned a lot.
Just use what you have and expand as you need. Racks are cool once you're into the hobby enough to care about style points.
Personally I buy new cheap mini PCs and consumer networking stuff. That's another way to get bargains, since consumer hardware is so cheap. But I'm a bit jealous of the enterprise capabilities like proper managed switches or remote managed servers.
I do use Dell PowerConnect manged switches from EBay, but tbh for what I do, TP Link switches work really well and are cheap. They support VLANs, MAC Lans, SNMP, SSH/Web configuration, etc etc - basically fully managed switches. And they have been rock solid as 'edge' switches for each room in the house. The powerconnects basically work as top-of-rack switches for pulling all the different rooms/rack into 1 feed to the cable modem.
I picked up a HP managed switch that I thought I might use but it draws more power than the ThinkCentre M93p Tiny computers do and it's actively cooled w/ super noisy 1U fan(s) so I'm going to ditch it.
Personally I was looking to build a router so I ended up buying a fanless N100 based mini PC from Aliexpress (e.g.: search term is "N100 firewall appliance") and have been very satisfied with it so far (Proxmox homelab with OPNsense running as a VM).
$170, N100, 16GB LPDDR5, dual 2.5GbE, https://aoostar.com/products/aoostar-n-box-pro-intel-n100-mi...
$210, N100, dual 3.5" SATA, M.2 NVME, dual 2.5GbE, https://aoostar.com/products/aoostar-r1-2bay-nas-intel-n100-...
$250, Ryzen 5700U, https://aoostar.com/products/aoostar-cyber-amd-ryzen-7-5700u...
N100 sips power while performing as well as a 60w quad core Skylake.
I've found the TP Link stuff to be good for home switches. cheap, quiet, and basically fully featured/managed. Stuff like SNMP, VLAN/Mac Lan, port mirroring/monitoring, etc. Also SSH and web interfaces... I have 2-3 currently, 1 per room feeding back to the PowerConnects (NOISY) and haven't had a single issue in 3-5 years of use :-)
Thanks!
It depends on the generation of the gear. Stuff coming off-lease in the last 3 - 4 years has been of a generation where lower power draw became a major selling point. In Dell land, for example, the power draw differences are dramatic between the Rx10, Rx20, and newer series. (Basically, throw Rx10 servers in the trash. Rx20 servers are better but still not great.)
All depends on what you’re going for though.
And the heat is enough to make a room very uncomfortable. I started my home lab in my office, and it didn't take long to realize that was not an option. Luckily I was able to find a nook under the stairs where I could stash it, but the heat and noise are polar opposite of a productive office environment.
They got a managed switch, nice. It has VLANs, stacking, etc. nice. How many 48 port switches you can possibly stack in a home? You got a Sun ZFS7420. Great. That's a rack with two controllers with 1TB RAM each. That thing screams. An entry level 7320 is maybe workable, but you can't do 7420 at home. Same for HPE/Dell/Netapp storage boxen.
Doing Infiniband is home is another thing I can't understand. Fiber cables are extremely expensive. Copper is range limited and thick, plus unless you do HPC, its latency and speed doesn't mean much. Go fiber Ethernet. Much more practical.
I understand the curiosity and willingness to play, but it's not that practical to play with these at home (noise, space, heat, power wise). I also buy high end consumer hardware. TP-Link has 4/8 port smart managed SOHO switches for example, they even have PoE. Power your Raspi cluster via PoE with less cables and get some respectable computing power with no noise. Neat! No?
Dell's Rack servers are generally not noisy at idle, but a 2U server under some real load is unbearable. What's funny that an Intel N95 has the same power with an i7-3770K with better SIMD support while being 4x more efficient, and that thing is not slow by any means.
A 5m, 200gbps cable is $1,149.00 a pop [0].
Mind you, this is neither the longest, nor the fastest cable.
[0]: https://esaitech.com/products/mellanoxmfs1s00-h005v-200gbps-...
More cables: https://esaitech.com/search?type=product&options%5Bprefix%5D...
With 3d printed powerbrick holders on the sides and a switch with a 10g uplink so all servers can use the full 1g nic.
Warms my home office in the colder months and can be put in the garage on the warmer months.
Photo front: https://pictshare.net/qi7zv2.png
Photo back: https://pictshare.net/llloiw.png
Photo of the 3d printed power brick holders: https://pictshare.net/kgffy6.png
[edit] Specs:
- 80 Watts during normal load
- 50 cores
- 9TB nvme storage
- 300GB RAM
- 3 of them are running proxmox
- 3 of them Docker swarm
The advantage with a vertical rack for me was space savings. Key parts of my home lab are near the low voltage panel (lucky to have in newer house) which does have foot traffic from other household members. Much less risk of someone bumping into the rack.
Actually, Asus has taken over NUCs from Intel and they are releasing fanless machines, so now there is not even the need to mod them.
I run a half dozen Lenovo ThinkCentre tiny PCs I picked up off eBay super cheap. Sure, I could go buy some rack mounts for them all… to the tune of $750. It would look really nice. That’s about it. If I’d decided that was a “requirement” I never would have done any of this.
For anyone considering diving in to any of this stuff, don’t let the pursuit of “perfect” stop you before you’ve even started.
My PCs are quite content sitting in a pile with some scraps of wood shoved between to help with airflow.
I got kids. So I got a lot of toys in the living room. The small office I got, does contain a printer, a PC, a space for my laptop plus dock, a server, a switch, and a small cluster. But what I don't have is more space. I don't have space for a rack.
Furthermore I've been replacing industrial grade fans (in switches, desktop, the cluster and the like) which make a lot of noise (and cool very well) with Noctua which are expensive and have less CFM but are silent enough.
I posted this in a sibling comment, but it doesn't really have to be expensive regardless. I got started with the homelab stuff with a partially broken laptop. Considering that that laptop was about to go into the trash, I consider that (sort of) free to get started, but even if you don't have an old laptop lying around you can purchase a used "Craigslist Special" for basically nothing, generally less than $200, and often much much much less because you probably have a friend or coworker that has an old laptop lying around that they'd be happy if you took off their hands. Old laptops are kind of in this frustrating category of being "too good to throw away, but not good enough to use", so people will often hoard them inadvertently. Additionally, they're extremely ubiquitous; most households in the US have at least one laptop, and often at least one per person.
Even rack mount servers (used) cost very little, at least upfront. My server with 128gb of RAM and 24 cores was $150, but of course the power draw is where most of the cost for that comes from, so for those you could argue that they're a rich man's hobby.
ETA:
For those interested, the listing I ordered from a few years ago on Amazon appears to have mutated into a different item, but this one appears to be an identical product. I think for the price it's actually a really decent keyboard. I'm able to get about 120WPM on MonkeyType with it, for what it's worth.
https://www.amazon.com/Mechanical-Keyboard-Ultra-Slim-Anti-G...
Sure you can spend thousands on racks of servers and UPS/generators but you can also spend less than $500.
Personally I rent a unit in a quadplex and have:
Old 42" rack - $50 from old employer
10 year old HP Microserver G7 for storage - $250 new
repurposed old desktop virtual machines - Free
Cheap dumb switch - $50There's no definition like that, but it happens to turn out that way.
"Deals" on used enterprise are IME dreadful unless you can get in on your own employer offloading old stuff, which requires you to a) even have an onprem setup at all and b) be friendly enough with the IT/NOC people that they give you a tip when it's going to happen. (For me b seems to happen naturally so it's really all on a).
eBay is a fools game, terrible prices. Craigslist is usually the same sellers too. Maybe some people live behind an Amazon datacenter or something, I unfortunately live in a city where the only reason we have Amazon 2-day shipping is because we're a major shipping hub. And anything that does resemble a good deal is ancient Opterons that are literally e-waste these days.
So anyway, sorry, my whole point here is that most people are actually going to be running a ton of old enterprise stuff that is actually rackmountable with good sliding rails. You're either building whiteboxes in a 4U rosewill case or you're putting stuff on rack-mountable shelves. The rosewill cases theoretically have sliding rails but they're awful, just awful. So you're either using shelf rails or literal shelves which tends to throw off efficient utilization of the U's available in the rack - shelf rails usually result in a dead U either above or below them, and shelves obviously have similar effects.
So yeah. As someone who owns a rack (4-post open rack), if you asked me, I'd say just go buy one of those big heavy duty Lowe's shelf units and ask for extra shelf inserts so you can have more, but less-tall shelves. Boom, you now have something that'll hold anything and is way better than fucking around with rack nuts and shelf rails and stuff.
Love homelabs in general, though.
These days I've been replacing the guts of my old servers with mini-itx setups for the most part. It does what I need and lets me shove 2 or 3 nodes into a 2u case so I can keep the aesthetic of "a personal rack".
Honestly I don't mind wasting 1 or 2U between machines because the tops of server cases function as great "temporary" shelves.
I do agree the meme of used enterprise deals is completely overblown, unless you actually want 15+ compute nodes and are prepared to suffer power, heat, and noise for the density. Honestly I wish they made those Rosewill cases in 5U so I could put any full height cooler in there instead of being somewhat restricted.
- shelf with two mini systems - power strip with individual switches - power strip always-on outlets front/back - ethernet switch - "broom" thing to feed cables through - many-port USB 3 hub
other systems are under the desk.
It's really convenient. I have hooked some systems to power switches, others are always on. It is organized, takes up space but doesn't fall over, and gives me desktop access to ports if I need them.
That said, many years ago I had a server setup where I really organized everything, zip ties for all cables for neatness, etc. I turned out to be a pain every time I rewired something (cut zip ties, etc). I eventually learned there's a neat and organized vs flexibility balance you have to maintain.
I got a few recently, and they're just great, particularly in terms of power usage. I hooked mine to tasmota-powered smart plugs, and they idle at something like 6W ... Granted, I always tune hardware (from bios config) and software (tuned profile) for low power... But long story short, my nucs rarely spike over 30W.
Literally half of one of those old tungsten-based lightbulbs.
And I can update the ram and use ECC.
Not bad all in all !
Still, I got a Dell SFF with an I5 and 16gb of ram for less than 100€ — updated it with a 4 2,5gbps NIC, will be good for OPnsense... If I don't switch to a Melanox Connect 3.
I don't really need that much horsepower (My stuff is like 90% idle - git server, etc.) but I _really_ appreciate the lower noise and power usage compared to my old Xeon blades
plenty of power for my needs, silent, low power ... and CHEAP! I especially love the large RAM capacity as it lets me run a lot of docker/vm/etc without swapping. Most of my stuff is idle like 90% of the time (stuff like git servers, etc) so the CPU is fine for me.
I've had mine for a few years now (I had the first gen..) and never any problems!
https://news.ycombinator.com/item?id=32619740
:)
Glad to hear they never stopped doing what they were good at, sounds like they've gotten even better!
I really hope more people hear of the H2/H3 stuff...I'd love for Odroid to keep making/improving them! :-D
Amazing how much a kind gesture sticks with you! Glad to see stuff like that these days :-D
[0] https://www.asus.com/us/content/nuc-overview/ [1] https://www.howtogeek.com/say-goodbye-to-intel-nuc-as-asus-t...
You can buy old used laptops (especially with broken monitors) for basically nothing (or maybe literally nothing if you already have one, or you have a friend of family member who has one they're willing to donate). If your goal is just to use it for Plex, it will work fine, and the power draw on laptops (particularly when idle) isn't that much more than a NUC.
I use a proper rack mount server now largely because I thought they were neat, but I think for most people an old laptop is fine.
I also think it's good to prevent ewaste. There's a lot of raw computing power that just ends up in landfills. It's almost certainly better for the environment to get as much use out of existing stuff as possible than it is to buy new stuff. I can't imagine it's good for the world for a bunch of old lithium batteries to lie dormant.
If I can find a good deal on a dozen old laptops, I might honestly put this plan into action.
That being said, when I finally got around to rackmounting and upgrading some of the other hardware in my lab, this "beginner"'s guide was really helpful.
It's not the most sophisticated setup on earth, but it works great, it's dirt cheap, and it's apartment-friendly.
I used to use a half rack full of Xeon based Dell and HP blades - but the blade chassis are huge/heavy and the power usage was HUGE!
Edit: Oh shit...just checked, it does NOT have a PCIE slot..I was sure it did :-(
Granted, this is one server. If you have a bunch of them running at high load all at once then of course that can get pretty pricey power-wise.
I actually ran 3-4 extra circuits to the computer room to run it all...
In Sydney, Australia running your server just at idle would cost just over AUD $50 a month assuming it is powered by the grid.
For about AUD $400 you could replace it with a 7 or 8 series Intel i7 mini PCI and throw a 10Gbe NIC in it and that thing will cost you AUD $7 a month to run.
It gets tricky though if you want ECC RAM. Most of the cheaper low power systems that are available will take UDIMMs and that becomes prohibitively expensive if you need a decent amount of RAM.
Well, your monthly power bill would be much higher than needed with either those light bulbs or a server like that.
Those light bulbs can be cheap to replace with more efficient variants. This was already true approx 30 years ago, back in the 90s.
I got solar power. Without that, I wouldn't be having a Xeon server right now.
Don't get me wrong, to each their own! That is the fun thing with homelab. You can optimize for power, price, noise, size, heat, or a combination of these. Then you can decide to have a couple of machines dedicated or have one large VM. Then you pick the OSes.
For example, right now I am settled for my homelab server. I got a cluster to play with, too. I got a 10 gbit switch (which I only put on if I need the speed, otherwise I use a second hand Brocade with 10 gbit SFP+ uplink). Loads of fun. But my router is just an Edgerouter-Lite. EdgeOS is dead, and it doesn't have failover nor does it run a VM. If I run it in bridge mode (which I do now) and I use Wireguard it doesn't saturate 1 gbit at all. So I am looking for an alternative which is a bit future proof. Ie. with SFP+ because I will get fiber instead of DSL eventually.
I’ve been debating trying to find a smaller, low power computer to reduce energy; it’s hard to find a small one that have 10GbE or that would take a pcie card. Maybe I can find a Thunderbolt one.
Yeah, Thunderbolt would work. I use LAG on my NAS with 4x 1 gbit, I got some devices with USB-C to 2.5 gbit NIC, these are cheap (30 EUR or so). I'd then use that managed 10 gbit switch which I put off by default. If I'd need the speed, I'd put it on. Thunderbolt is even better (way faster and cheaper, multipurpose, though you'd need to set up TCP/IP over USB), but I prefer AMD over Intel.
If you want a TB4 one with Intel, there's many Intel NUCs available, very cheap on Ali. Many of the ones I saw not from Ali are just rebrands (some, admittedly, with Coreboot).
I was looking into some of those single-purpose computers, some of which will sometimes have a PCIe slot.
I don't need Proxmox, tons of containers or anything. If I do, I can always throw a N95 into the mix with ample RAM and storage.
Additionally, one challenge I have with all of these is that none of them have anything like a BMC. If you have more than one system, moving peripherals around sucks, and KVM technology is kind of iffy these days. It seemed easier to just jam an old Supermicro motherboard that had IPMI enabled in a reasonable ATX case with a lower power Xeon variant, and call it a day.
Personally I turn IPMI off. The difference at idle can be up to 20w. Not to mention the outdated interfaces are a sercurity risk
Just do not put CPUs with >65W there. Or hungry PCI things.
[1] https://thin.computer/index.php/2024/03/08/self-hosting-my-s...
It was so good actually that I've made it my main PC. I already had 3 monitors and this SFF just worked fine with them. No games of course.
I also have Precision Tower with Xeon/64GB ECC RAM and 1 native NVMe slot. I use PCI-to-NVMe adapters there (2 slots), it works well with ZFS mirror.
Have you considered using a NAS as your homelab? I run a few containers (no VMs yet, although I can) and just do that on my QNAP.
The TBS-h574TX (https://www.qnap.com/en-us/product/tbs-h574tx) has five E1.S/M.2 slots. The HS-453DX-8G has two SATA 3.5-inch drive bays and two M.2 2280 SATA SSD slots (https://www.qnap.com/en/product/hs-453dx), and the TBS-h574TX. Or you can just throw SATA SSDs into any NAS.
That said, I was running the Homelab on effectively a single Unraid hyperconverged NAS/compute layer before then, and that worked wonders for quite a long time. I wouldn't consider Unraid my first choice now, given the advancements in TrueNAS SCALE in the time since I chose it as the solution, but the concept of NAS + Container Orchestrator as hyperconverged homelab today is quite attractive.
Yeah, that's what I was looking at Proxmox (8.x) for recently as well, but it never made it through basic qualification testing. VM migration's would randomly hang forever about 25% of the time. Definitely not a case of bad network interconnect either. :(
I'm using Ryzen 5000 series cpus on all the test gear though, and recently saw a mention that it may have been a known problem that's fixed. But not super keen on wasting more time.
What sort of cpus are your hosts using, and are you using your VMs with Ceph as their storage? :)
That said, I do have a NAS that I can provide both NFS and iSCSI storage to, so I imagine if I run NFS as the backing storage, I shouldn't have to worry about clustered file systems. Each compute does have local storage, but the plan is to provide that via a CSI for workloads that don't need to be highly available or where fault tolerance isn't a concern.
Can proceed with more testing now. :)
My problem is that none of them have an internal power supply. They all require a power adapter that's decidedly less easy to source than an IEC cord. I have had 3 separate experiences where I give someone instructions to grab a box and the power adapter... and they show up with the box and without the power adapter (or with a power adapter in a box of similar power adapters and no volt/amp/size rating on the box to disambiguate). In short, the internal power supply is something I actually care about.
The best bet might be to wait until they've all finalized on USB-C.
* https://eshop.macsales.com/shop/ssd/owc-accelsior-4m2 (4x NVMe drives)
* https://eshop.macsales.com/shop/ssd/owc-accelsior-8m2 (8x NVMe drives)
I'd personally get them without drives included, as the OWC drives seem pretty expensive.
I did not know just how much heat a 16tb disk can put out up until that point though.
I'm annoyed at the lack of ECC-capable sub-ITX NAS boards. I have a Helios4, but I've no idea what I'll migrate to when that dies.
then a better name than homelab would be your "NUC/NOC-nook".
with regard to TFA, I don't trust "labs" with neat wiring.