Home Lab Beginners guide
linuxblog.io
linuxblog.io
I personally have 4 boxes stacked in a corner that are connected to a "rack", which is just two rack side panels bolted to the back of a workbench that I screw components into, connected to NAS and multiple Raspberry Pi's on a shelf and I really like the mess and have learned a lot.
Just use what you have and expand as you need. Racks are cool once you're into the hobby enough to care about style points.
Personally I buy new cheap mini PCs and consumer networking stuff. That's another way to get bargains, since consumer hardware is so cheap. But I'm a bit jealous of the enterprise capabilities like proper managed switches or remote managed servers.
With 3d printed powerbrick holders on the sides and a switch with a 10g uplink so all servers can use the full 1g nic.
Warms my home office in the colder months and can be put in the garage on the warmer months.
Photo front: https://pictshare.net/qi7zv2.png
Photo back: https://pictshare.net/llloiw.png
Photo of the 3d printed power brick holders: https://pictshare.net/kgffy6.png
[edit] Specs:
- 80 Watts during normal load
- 50 cores
- 9TB nvme storage
- 300GB RAM
- 3 of them are running proxmox
- 3 of them Docker swarm
There's no definition like that, but it happens to turn out that way.
I run a half dozen Lenovo ThinkCentre tiny PCs I picked up off eBay super cheap. Sure, I could go buy some rack mounts for them all… to the tune of $750. It would look really nice. That’s about it. If I’d decided that was a “requirement” I never would have done any of this.
For anyone considering diving in to any of this stuff, don’t let the pursuit of “perfect” stop you before you’ve even started.
My PCs are quite content sitting in a pile with some scraps of wood shoved between to help with airflow.
The advantage with a vertical rack for me was space savings. Key parts of my home lab are near the low voltage panel (lucky to have in newer house) which does have foot traffic from other household members. Much less risk of someone bumping into the rack.
- shelf with two mini systems - power strip with individual switches - power strip always-on outlets front/back - ethernet switch - "broom" thing to feed cables through - many-port USB 3 hub
other systems are under the desk.
It's really convenient. I have hooked some systems to power switches, others are always on. It is organized, takes up space but doesn't fall over, and gives me desktop access to ports if I need them.
That said, many years ago I had a server setup where I really organized everything, zip ties for all cables for neatness, etc. I turned out to be a pain every time I rewired something (cut zip ties, etc). I eventually learned there's a neat and organized vs flexibility balance you have to maintain.
Actually, Asus has taken over NUCs from Intel and they are releasing fanless machines, so now there is not even the need to mod them.
"Deals" on used enterprise are IME dreadful unless you can get in on your own employer offloading old stuff, which requires you to a) even have an onprem setup at all and b) be friendly enough with the IT/NOC people that they give you a tip when it's going to happen. (For me b seems to happen naturally so it's really all on a).
eBay is a fools game, terrible prices. Craigslist is usually the same sellers too. Maybe some people live behind an Amazon datacenter or something, I unfortunately live in a city where the only reason we have Amazon 2-day shipping is because we're a major shipping hub. And anything that does resemble a good deal is ancient Opterons that are literally e-waste these days.
So anyway, sorry, my whole point here is that most people are actually going to be running a ton of old enterprise stuff that is actually rackmountable with good sliding rails. You're either building whiteboxes in a 4U rosewill case or you're putting stuff on rack-mountable shelves. The rosewill cases theoretically have sliding rails but they're awful, just awful. So you're either using shelf rails or literal shelves which tends to throw off efficient utilization of the U's available in the rack - shelf rails usually result in a dead U either above or below them, and shelves obviously have similar effects.
So yeah. As someone who owns a rack (4-post open rack), if you asked me, I'd say just go buy one of those big heavy duty Lowe's shelf units and ask for extra shelf inserts so you can have more, but less-tall shelves. Boom, you now have something that'll hold anything and is way better than fucking around with rack nuts and shelf rails and stuff.
Love homelabs in general, though.
I got a few recently, and they're just great, particularly in terms of power usage. I hooked mine to tasmota-powered smart plugs, and they idle at something like 6W ... Granted, I always tune hardware (from bios config) and software (tuned profile) for low power... But long story short, my nucs rarely spike over 30W.
Literally half of one of those old tungsten-based lightbulbs.
And I can update the ram and use ECC.
Not bad all in all !
Still, I got a Dell SFF with an I5 and 16gb of ram for less than 100€ — updated it with a 4 2,5gbps NIC, will be good for OPnsense... If I don't switch to a Melanox Connect 3.
You can buy old used laptops (especially with broken monitors) for basically nothing (or maybe literally nothing if you already have one, or you have a friend of family member who has one they're willing to donate). If your goal is just to use it for Plex, it will work fine, and the power draw on laptops (particularly when idle) isn't that much more than a NUC.
I use a proper rack mount server now largely because I thought they were neat, but I think for most people an old laptop is fine.
I also think it's good to prevent ewaste. There's a lot of raw computing power that just ends up in landfills. It's almost certainly better for the environment to get as much use out of existing stuff as possible than it is to buy new stuff. I can't imagine it's good for the world for a bunch of old lithium batteries to lie dormant.
If I can find a good deal on a dozen old laptops, I might honestly put this plan into action.
That being said, when I finally got around to rackmounting and upgrading some of the other hardware in my lab, this "beginner"'s guide was really helpful.
It's not the most sophisticated setup on earth, but it works great, it's dirt cheap, and it's apartment-friendly.
I used to use a half rack full of Xeon based Dell and HP blades - but the blade chassis are huge/heavy and the power usage was HUGE!
Granted, this is one server. If you have a bunch of them running at high load all at once then of course that can get pretty pricey power-wise.
In Sydney, Australia running your server just at idle would cost just over AUD $50 a month assuming it is powered by the grid.
For about AUD $400 you could replace it with a 7 or 8 series Intel i7 mini PCI and throw a 10Gbe NIC in it and that thing will cost you AUD $7 a month to run.
It gets tricky though if you want ECC RAM. Most of the cheaper low power systems that are available will take UDIMMs and that becomes prohibitively expensive if you need a decent amount of RAM.
Well, your monthly power bill would be much higher than needed with either those light bulbs or a server like that.
Those light bulbs can be cheap to replace with more efficient variants. This was already true approx 30 years ago, back in the 90s.
I got solar power. Without that, I wouldn't be having a Xeon server right now.
Don't get me wrong, to each their own! That is the fun thing with homelab. You can optimize for power, price, noise, size, heat, or a combination of these. Then you can decide to have a couple of machines dedicated or have one large VM. Then you pick the OSes.
For example, right now I am settled for my homelab server. I got a cluster to play with, too. I got a 10 gbit switch (which I only put on if I need the speed, otherwise I use a second hand Brocade with 10 gbit SFP+ uplink). Loads of fun. But my router is just an Edgerouter-Lite. EdgeOS is dead, and it doesn't have failover nor does it run a VM. If I run it in bridge mode (which I do now) and I use Wireguard it doesn't saturate 1 gbit at all. So I am looking for an alternative which is a bit future proof. Ie. with SFP+ because I will get fiber instead of DSL eventually.
I actually ran 3-4 extra circuits to the computer room to run it all...
Edit: Oh shit...just checked, it does NOT have a PCIE slot..I was sure it did :-(
I don't need Proxmox, tons of containers or anything. If I do, I can always throw a N95 into the mix with ample RAM and storage.
Additionally, one challenge I have with all of these is that none of them have anything like a BMC. If you have more than one system, moving peripherals around sucks, and KVM technology is kind of iffy these days. It seemed easier to just jam an old Supermicro motherboard that had IPMI enabled in a reasonable ATX case with a lower power Xeon variant, and call it a day.
Just do not put CPUs with >65W there. Or hungry PCI things.
Personally I turn IPMI off. The difference at idle can be up to 20w. Not to mention the outdated interfaces are a sercurity risk
[1] https://thin.computer/index.php/2024/03/08/self-hosting-my-s...
It was so good actually that I've made it my main PC. I already had 3 monitors and this SFF just worked fine with them. No games of course.
I also have Precision Tower with Xeon/64GB ECC RAM and 1 native NVMe slot. I use PCI-to-NVMe adapters there (2 slots), it works well with ZFS mirror.
Have you considered using a NAS as your homelab? I run a few containers (no VMs yet, although I can) and just do that on my QNAP.
The TBS-h574TX (https://www.qnap.com/en-us/product/tbs-h574tx) has five E1.S/M.2 slots. The HS-453DX-8G has two SATA 3.5-inch drive bays and two M.2 2280 SATA SSD slots (https://www.qnap.com/en/product/hs-453dx), and the TBS-h574TX. Or you can just throw SATA SSDs into any NAS.
My problem is that none of them have an internal power supply. They all require a power adapter that's decidedly less easy to source than an IEC cord. I have had 3 separate experiences where I give someone instructions to grab a box and the power adapter... and they show up with the box and without the power adapter (or with a power adapter in a box of similar power adapters and no volt/amp/size rating on the box to disambiguate). In short, the internal power supply is something I actually care about.
* https://eshop.macsales.com/shop/ssd/owc-accelsior-4m2 (4x NVMe drives)
* https://eshop.macsales.com/shop/ssd/owc-accelsior-8m2 (8x NVMe drives)
I'd personally get them without drives included, as the OWC drives seem pretty expensive.
I did not know just how much heat a 16tb disk can put out up until that point though.
I'm annoyed at the lack of ECC-capable sub-ITX NAS boards. I have a Helios4, but I've no idea what I'll migrate to when that dies.
then a better name than homelab would be your "NUC/NOC-nook".
with regard to TFA, I don't trust "labs" with neat wiring.
Proxmox makes it very easy to have multiple containers and VMs on a single hardware device. I started by just virtualizing the one big Ubuntu system. Even that has advantages: backups, high availability, etc. But now I'm starting to split off services into their own containers and it's just so tidy!
Most recently I set up a Windows VM with PCIe GPU passthrough for streaming (Moonlight/Sunshine) games to the various less-capable machines in my house, and it works so well that my actual gaming PC is sitting in the corner gathering dust.
My only complaint is I wish there was a cheaper paid license. I would love to give back to them for what I'm getting, but > $100/yr/cpu is just too much for what amounts to hobbyist usage. I appreciate the free tier (which is not at all obnoxious about the fact that you aren't paying), but I wish there could be a middle ground.
I don't use proxmox anymore at home (1) but as an user I would have loved to pay, say, $5/month and just get access to the updates repository (the stable ones) or maybe something similar. $100/year/cpu was just too much for me as an hobbyist.
[1]: not because I don't like it, for life reasons I had to downsize that hobby a bit and I don't need proxmox anymore.
Anyone from Proxmox reading? Could it be that you're leaving money on the table here?
No, I do not think we leave much money at the table.
In short, we're targeting enterprise users with a mix of a (soft) stick (e.g. pop-up to note that one isn't having the most production ready experience) a carrot (way better tested updates and depending on the level also enterprise support). Homelab users aren't in the target of our subscription services at all, and if we'd target them with cheap prices that would be just misused by companies too, we know this for a fact because the project is over 15 years old, and there was a lot tried out before getting to the current design. And while there might some protection mechanisms we could set up, we rather avoid DRM'ifying Proxmox projects and avoid wasting time playing cat-and-mouse games with entities trying to abuse this.
Note though, that we still cater to the homelab in other ways even if it isn't our main target audience, like being very active on our community forums for all users, or simple having 100% FLOSS software, no open-core or other, in my opinion rather questionable, open source models. And the price of a pop-up or using the still very stable no-subscription repository is IMO also quite small compared to the feature on gets 100% for free.
If one wants to contribute to our project but either cannot, or does not want, to afford a subscription, then I think helping out in the communities, submitting elaborate bug reports and thought out feature requests, spreading gospel at the companies they work with/for is not only cheaper for them, but also much more worth for the project.
ps. The actual license is the AGPLv3, which is always free, what we sell are subscriptions and trainings for additional services.
Proxmox' official solution for this is Ceph but that seems like overkill for a small home setup.
I also hear some people just go with TrueNAS or Unraid as their bare-metal base and use it for both storage and as a hypervisor, which makes sense. I might have to try the Linux version of TrueNAS now that it's had some time to mature
I also rely on my Intel processor's built-in Quicksync for HW transcoding. Is there any trouble getting that to run through a VM, except presumably sacrificing the local Proxmox TTY?
[1] As in the one with the files on, not VHDs. I default to keeping those separate in the first place, but it's also necessary since I can't afford to put all data on SSDs nor tolerate OS roots on HDDs. Otherwise a single master ZFS array with datasets for files and zvols or just NFS for VM roots would probably be ideal
[2] Which I hear is considered more reliable than individual SATA passthrough, but has the caveat of being more coarse-grained. I.e. you wouldn't be able to have any non-array disks attached to it due to the VM having exclusive control of all its ports
Yep, I do this and it's very simple to give LXCs access to parts of my storage array.
Unraid is especially meant for that. But TrueNAS Scale is also good, it has Docker and Kubernetes support too.
There is also openmediavault, it's a bit more Linux-y than the Unraid or TrueNAS Scale but a lot of people love it
I use xcp-ng rather than Proxmox.
0: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_mig... & https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_guest...
1: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_how_i...
2: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapte...
I just wish there was something like Unraid's Docker support but in a FOSS distro.
Proxmox with LXC is just not the same. Unraid is good, love the UI and easyness too but the proprietary paid Linux is just something I don't like.
I like LXC alright but I do miss the ecosystem of DockerHub.
https://unraid.net/blog/pricing-change
disclaimer - I'm not an Unraid user.
I've mostly just been defaulting to k3 since I use k8s at work so lots of my knowledge maps over.
I find myself referring to my git repo frequently as a source of documentation on what's installed and how the systems are set up.
I totally agree. It (nix and nixos) is/are such a pain so often. But, wow... when you get things to work it is glorious.
Quickly and easily syncing system/user/application configurations across multiple devices is incredible. The declarative approach means that the often laborious effort it takes me to get things to work pays off when I finally figure it out and realize I never have to do that again - and since it's declarative, I now have a documented functional configuration/example to grow and expand from.
My current project is to learn Nix development environments by building "Linux from Scatch" from Nix. I am making progress slowly, but every success is cemented and replicatable.
LLMs definitely help distill the sparse/scattered documentation, but I am finding a good amount of grit is still required.
All in all, I am very much enjoying Nix and hope to utilize it more in my homelab.
It's annoying enough that I won't use nix personally. I've used it at work so I'm quite knowledgeable with it.
The problem I've realized is that for many core Nix developers, the source is the documentation. Which doesnt scale.
DietPi [1] has one massive (15KLoC) bash script [2] for system config.
[2] https://github.com/MichaIng/DietPi/blob/master/dietpi/dietpi...
I've been hearing more and more from people switching over to NixOS, so hopefully the docs will get better as well
Getting a basic system going seemed about as easy as Arch (if you consider Arch easy). But adding flakes and home manager to the mix is where things have run off the rails a bit for me.
I think a lot of the steepness of the learning curve comes from trying to go all in on flakes/home manager up front, and a lot of the online discussions and quick starts being oriented towards this goal. I’ve since scaled back my aspirations a bit and I’m focusing on just understand core NixOS with a monolithic config file until it makes more sense to my brain.
If you haven’t get, check out vimjoyer’s channel on YouTube. His content has been the most helpful for me by far, and it got me a basic setup going that I’m now iterating on. What I do love is that I feel like I can iterate with confidence. I have no idea what I’m doing for the most part but I know I can get myself back to a working state.
I think it will be worth the time investment in the long run, but yeah, it’s been a bit weird and the lack of a canonical approach makes it hard to break into the ecosystem.
You might try to generate your docker containers using nix as a first step. Not sure if it will be more approachable for you, but at least its 1 less new thing to adjust to.
Someone is going to chime in with "boo yaml" and sure, but a person interested in running a homelab is almost certainly a person who manages a bunch of servers for work too and there's no escaping cloud-init, Ansible, Kubernetes, or something using yaml that you're not going to have a choice about. Assuming the point of a homelab is at least partly to practice what you do on the job without the possibility of destroying something owned by someone else (which is what this guy says his reason was), you may as well get used to the tools you're going to have to use anyway and not make your home setup too much different than the environment you manage at work.
Of course, I know a lot of people on Hacker News are not like me and this guy and use "homelab" to mean they want to self-host media servers, chat, and photo sharing and what not for their family and friends, not have a mini data-center as a practice ground for the real data center they manage in their day job.
The only issue is remotely deploying Nix configs. The only first-party tool, nixops, is all but abandoned and unsupported. The community driven tools like morph and deploy-rs seem promising, but they vary in terms of Flakes support and how much activity/longevity they seem to have.
[0] https://blog.janissary.xyz/posts/nixos-install-custom-image
Disclaimer: I am a relatively active contributor
I avoid this by running Gentoo Linux on my Linux machines. I've heard people say good things about Arch, but I've never used it.
Ubuntu is just bad... I say this as someone who used Ubuntu for a few years waaay back when (and was _really_ pleased that I finally could recommend a reliably, easy-to-use Linux distro to nontechnical folks), but swore off of it after the second or third in-place upgrade (in a row) that left my system nonfunctional in ways that required the sysadmin knowledge I'd gained from going through the well-documented Gentoo setup process (and from tinkering over the years) to repair.
Friends don't let friends use Ubuntu.
Gentoo on ZFS is kinda nice.
My home lab journey was:
manual installs -> Ansible -> Docker + Docker Compose -> Kubernetes (I told you I loved complicated things)
I kept everything source controlled. This repo documents my long journey: https://github.com/shepherdjerred/servers
Docker and Docker Compose are nearly perfect. Here's an example of what that looked like for me: https://github.com/shepherdjerred/servers/blob/839b683d5fee2...
I mostly moved to Kubernetes for the sake of learning. Kubernetes is very cool, but you have to put in a large amount of effort and learning before it becomes beneficial.
Ansible is nice, but I didn't feel like it met all of my needs. Getting everything to be idempotent was quite hard.
I didn't mind manually installing Docker, configuring fstab, etc., since it was such little work.
Because:
> "wait, how did I fix that thing 6mo ago?"
Such things just don't happen. That thing (RHEL/RockyLinux) just doesn't break. I do update software once every six (or more) months, reboot and it just works.
I have notes on how I did stuff, if necessary. I used to keep ansible playbooks but they were just an overkill. Does it even make sense to write a playbook for a system that I'll reinstall in three or four years ? Probably not.
One of my main systems at home has been running RHEL 8 for 3-4 years, still happily getting updates. I plan on replacing the whole box, with another one running RockyLinux 9.
And I'm moving from RHEL to RockyLinux because I just don't want the yearly annoyance of having to renew the Developer Subscription license.
That's how reliable it is: I have to look up my notes once a year, and it's too much.
I run most of my stuff in containers anyway, and podman is just great.
Yea, sure, except what if the one-time configuration was in a Nix file that will continue being exactly as useful for years?
Folks applying the Ansible mindset to NixOS are missing the point. NixOS is *not like a runbook*. It's declarative, it's prescriptive, it's not some god-forsaken state machine that tries to paper over the horrors of modern computing. It IS the definition of my computers, not some glorified pile of scripts and conditional checks encoded in fking jinja. The same as it's been mostly for 8 years now.
Still useless. Software change, the configs i make today might not make sense or even work in 3-5 years.
Nix and stuff (ansible etc) only make sense if you have to reinstall your systems multiple times in a single week (fine if your hobby is reinstalling linux on your laptop).
But if you do a full reinstall once every five years then it’s largely an exercise in futility.
Y'all don't know what you're talking about and once again the Ansible comparison belies in.
>reinstall once every five years then it’s largely an exercise in futility.
You are as wrong as you think your are right, I'd know I've been using NixOS for 8 years. You're just wrong.
Idk, this is why it's hard to talk about nix with people that think they know it but then also openly doubt things that you will one day take for granted if you take the time to learn it.
Seriously as someone who has used ansible a lot and nix a bit, I'd take NixOS over Ansible+Debian/Fedora any day.
For a few years I put mine inside an IKEA FRIHETEN sofa (the kind that just pulls†).
Pros:
- easy access
- completely invisible save for 1 power wire + 1 fiber (WAN) + 1 eth (LAN)††
- structure allows easy cable routing out (worst case you punch a hole on the thin low-end)
- easy layout/cable routing inside
- noise reduction for free
- doubles as warming in winter a seated butt is never cold
- spouse enjoys zero blinkenlights
- spouse didn't even notice I bought a UPS + a disk bay
Cons:
- a bit uncomfortable to manipulate things inside
- vibrations (as in, spinning rust may not like it) when sitting/seated/opening/closing
- heat (was surprisingly OK, not worse than a closet)
- accidental spillage risks (but mostly OK as design makes it flow around, not inside the container, worst case: put some raisers under hardware)
- accidental wire pull e.g when unsuspecting spouse is moving furniture around for housecleaning (give'em some slack)
†† that I conveniently routed to a corner in the back then hidden in the wall to the nearest plugs, so really invisible in practice.
One of the reasons I didn't do this was I didn't want a fire with sofa fuel right above/around it. I wasn't concerned about the servers, but a bit about the UPS. Sheet metal lining of the area, with good venting if the UPS battery type could leak gases, I'd feel a bit better about, but too much work. So the gear ended up away from upholstery, in rack/shelving, where I could keep an eye on it.
I would be really worried about ventilation and heat / fire risk.
Reminds me of the lack rack.
Although I did some yelling^Wsitting test and found it to be surprisingly impervious it was not so much to the furniture moving scenario.
That, and the UPS being the unlikely-but-real fire hazard. Old man printers-and-shotguns, y'know.
(disclaimer: I'm an internet rando, not a fireman, seek professional guidance)
But then again it's going to be a fire hazard anywhere except in a fireproof enclosure: it may give warm fuzzies that it's kept in sight but a) these things pack a ton of energy and would burn fast and long; b) fire propagates stupidly quick in residential locations; c) you're not looking at it 24/7, far from it.
In a way, ok the whole thing is fuel surrounding it, but this fuel first needs to be consumed for the fire to fully break out. IKEA products are also already designed with some form of fireproofness in mind (e.g textile and foam won't burn as easily as you'd think). It won't resist long but it may comparatively give a few seconds delay before it makes it worse and breaks out to the whole room.
This time could be improved by lining the interior with fire retardant panels, including the underside of the sitting area. Given the roomy boxy shape of it it should be quite easy to do. If done right it could even be near-airtight, so that fire could delay/choke itself due to lack of O2. This would need active ventilation for thermals, but it could shut itself down on a reasonable heuristic indicating trouble. Heck now I'm thinking automatically injecting CO2/spraying water droplets to further quench. Oh, an engineer's imagination running wild.
The most recommended - and much cheaper - investment though would be a fire/smoke detector, as whether in sight or not, with such fires the most sensible general recommendation one could make would be:
a) be alerted as early as possible, preferably before any flame/smoke is visible.
b) immediately get out, like, RIGHT NOW. Think after. Life is the one thing of true value that cannot be replaced.
I'm not kidding. You have ~30s tops from the beep to get the whole family out. After that smoke is irrevocably incapacitating and flashover is only a short minute away.
Watch this: https://m.youtube.com/watch?v=JGIICiX2CNI&t=141
Then watch it again, roleplaying yourself having to be woken up by the alarm, then having to navigate through smoke, heat, and panic to get the spouse and kids out. These 30s are going to be awfully shorter IRL than they already appear watching the video.
End of interlude.
Ventilation was passive yet OK, temps were higher that open air of course but good. A thing that helped is that I did not put your run-of-the-mill Xeon server blade in there, that would be dumb, although it would make for a nice synesthetic experience when watching Top Gun.
So I aimed for passive cooled stuff (ISP ONT+router, RPis, bunch of USB portable SSDs and HDDs) or active but low thermal requirements (headless Mac Mini, 5-disk bay, UPS). I monitored temps and they were well within parameters, I've seen much worse numbers in actual server rooms. The most noisy things were the 3.5" 7200rpm drives spinning up and seeking (constant rotation was inaudible) and the disk bay fan (lousy unlubricated bearing).
Given the side panel size, pretty sure one could hack active ventilation with big near-silent fans.
Everyone has some set of goals... low power, interesting processors, data ownership, HA, UPS/whole home UPS... and the home is the only common intersection of all these overlapping interests, and bits of software. Even more fascinating is the types of people it attracts, from professionals playing to people far outside the industry.
I have really taken the plunge and it recaptures some of the magic of the early internet (at least for me).
Serve the home, and level 1 on YouTube...
It intersects with proxmox community, home assistant community, all the NAS distros...
The spread out and over lapping nature of these groups really adds to the early internet vibe!
* Location: Sitting on a shelf in my basement office. Ventilation is okay, the WiFi is fine but not great.
* Hardware: An old PC I picked up at a neighborhood swap meet. I added some RAM taken from another old PC and bought a hard drive and WiFi card.
* Software: Debian stable and podman/podman-compose. All my useful services are just folders with compose files. I use podman-compose to turn them into systemd units.
If the stuff in the article is the kind of thing you're into, that's awesome, go ham! But you absolutely do not need to ever, and you certainly do not need to do it right away. I run a bunch of services that my family uses daily on this thing, and we use less than half the 16GB RAM and never get over 5% CPU usage on this old, ~free PC.
I've dabbled with VMs for test environments but generally just install on bare metal.
- My real problem is disaster recovery. It would take me forever to replicate everything, if I could even remember it all. Router configurations, switch configurations, NAS, all the various docker containers scattered across different vlans, etc. I mapped out my network early on but failed to keep it up to date over time. Is there a good tool to draw, document, and keep up-to-date diagrams of my infra?
- Backup and upgrading is also a persistent problem for me. I will often set up a container, come back to it 6 months later, and have no idea what I did. I have dozens of containers scattered across different machines (NUCs, NAS, desktops, servers, etc). Every container service feels like it has its own convention for where the bind mounts need to go, what user it should be run as, what permissions etc it needs. I can't keep track of it all in my head, especially after the fact. I just want to be able to hit backup, restore, and upgrade on some centralized interface. It makes me miss the old cattle days with VM clone/snapshot. I still have a few VMs running on a proxmox machine that is sort of close, but nothing like that for the entire home lab.
I really want to get to a point, or at least move towards a solution where I could in theory, torch my house and do a full disaster recovery restore of my entire setup.
There has to be something simpler than going full kubernetes to manage a home setup. What do you guys use?
My strat is having a deploy.ps1 script in every project folder that sets the project up. 80% of the time this is making the vm, rcloning the files, and installing/starting the service if needed. Roughly 3ish lines, using custom commands. Takes about 100ms to deploy assuming the vm is up. Sometimes the script gets more complicated, but the general idea is that whatever system I use under the covers, running deploy.ps1 will set it up, without internet, without dependencies, this script will work until the heat death of the universe.
After losing everything I reran the deploys (put them into a list) and got everything back.
I'm with you on some of the routing setups, mine aren't 100% documented either. I think my router/switch config is too complex honestly and I should just tone it down.
I personally use one single server (Fujitsu D3417-B, Xeon 1225v5, 64GB ECC and WD SN850x 2TB NVMe) with Proxmox and an OpenWRT Router (Banana Pi BPI-R3). The Proxmox draws ~12W and the OpenWRT around 4.5W in Idle and with NodeJS I can run MeshCommander on OpenWRT for remote administration.
Since I use ZFS (with native encryption), I can do a full backup on an external drive via:
# create backup pool on external drive
zpool create -f rpoolbak /dev/sdb
# create snapshot on proxmox NVMe
zfs snapshot -r "rpool@backup-2024-01-19"
# recursively send the snapshot to the external drive (initial backup)
# pv only is there to monitor the transfer speed
zfs send -R --raw rpool@backup-2024-01-19 | pv | zfs recv -Fdu rpoolbak
# create new snapshot
zfs snapshot -r "rpool@backup-2024-01-20"
# only send everything between 2024-01-19 and 2024-01-20
zfs send -RI --raw rpool@backup-2024-01-19 rpool@backup-2024-01-20 | pv | zfs recv -Fdu rpoolbak
This is also pretty helpful on rewinding virtual machines[1]. Recently my NVMe (a Samsung 980 Pro) died and restoring the backup via zfs (the backup shell command in reverse order from rpoolbak to rpool) took about 2 Hours for 700GB and I was back online with my server. I was pretty happy with the results, although I know that ZFS is kind of "experimental" in some cases, especially encryption.
1: https://pilabor.com/series/proxmox/restore-virtual-machine-v...
I also found a friend who I am convincing of the same issues and so we may try mirroring our homelab documentation/process so that the "bus factor" isn't 1.
I do have an ECC instance for data, to be backed up to a remote S3. I could torch it. It still remains a lab. I could just buy a new hardware and restore. But it's only about the valuable data (isolated to a couple of containers), not "restore of my entire setup".
So I spent time creating a robust backup solution. Every week I backup all the docker volumes for each container to an external NAS drive.
Then I also run a script which backs up my entire docker compose file for every single container in one gigantic compose file.
The benefit of this is that I can make tweaks to containers and not have to manually record every tweak I made. I know that on Sunday at midnight it all gets backed up
here's my home assistant deployment as a single file: https://github.com/pl4nty/homelab/blob/main/kubernetes/clust...
I deliberately nuked my onprem cluster a few weeks ago, and was fully restored within 2 hours (including host OS reinstalls). and most of that was waiting for backup restores over my slow internet connection. I think the break even for me was around 15-20 containers - managing backups, config, etc scales pretty well with k8s.
I’ve learned so much about containers, virtual machines and networking. Some of the self hosted applications like paperless-ngx [1] and immich [2] are much superior in terms of features than the proprietary cloud solutions.
With the addition of VPN services like tailscale [3] now I can access my homelab from anywhere in the world.
The only thing missing is to setup a low powered machine like NUC or any mini PC so I can offload the services I need 24/7 and save electricity costs.
If you can maintain it and have enough energy on weekends to perform routine maintenance and upgrades. I would 100% recommend setting up your own homelab.
[1] https://docs.paperless-ngx.com/ [2] https://immich.app/ [3] https://tailscale.com/
As an apartment-dweller, for a long time I had it in a closet. Once I moved it to living room, my solutions included:
* For discreet, IKEA CORRAS cabinet that matched my other furniture. I had rackmount posts in it before, but got rid of them because they protruded.
* For noise, going with gear that's either fanless, or can be cooled with a small number of Noctua fans. (I also replace the fans in 1U PSUs with Noctuas, which requires a little soldering and cursing.) I tend to end up with Atom servers that can run fanless in a non-datacenter except for the PSU.
* Also for noise, since my only non-silent server right now is the 3090 GPU server, I make that one spin up on demand. In this case, I have a command I can run from my laptop to Wake-on-LAN. But you could also use IPMI, kludge something with a PDU or IoT power outlet, find a way to spin down the 3090 and fans in software, make Kubernetes automate it, etc.
* For lights, covering too-bright indicator LEDs with white labelmaker tape works well, and looks better than you'd think. Black labelmaker tape for lights you don't need to see.
* For console, I like the discreet slide-out rack consoles, especially the vintage IBM ones with the TrackPoint keyboards. If I was going to have a monitoring display in my living room, I'd at least put the keyboard in a slide-out drawer.
* I also get rid of gear I don't need, or I'd need more than twice the rack space I currently have, and it would be harder to pass as possibly audiophile gear in the living room.
* For an apartment, if you don't want to play with the router right now (just servers), consider a plastic OpenWRT router. It can replace a few rack units of gear (router, switch, patch panel), and maybe you don't even need external WiFi APs and cabling to them.
What do you think the lowest budget or spending that I should have, given my “requirement”? Or would you say that vast.ai or similar websites would suffice?
Personally, for my changing needs, my GPU server is already on its 4th GPU, after I kept finding I needed more VRAM for various purposes. The RTX 3090 seems to be a sweet spot right now, and my next upgrade right now would probably be a 2nd 3090 with NVLink, or some affordable older datacenter/workstation GPU with even more VRAM. Other than the GPU, you just need an ordinary PC with an unusually big PSU, maybe a big HDD/SSD (for data), and maybe a lot of RAM (for some model loading that can't just stream it). https://www.neilvandyke.org/machine-learning/
Why do I have a home network? Like many here, I develop network applications. I'm not a network technician, but being able to truly understand how the various components of the internet work, like TCP/IP, DNS etc., is really useful and sets me apart from many developers. I also just like being in control and having the flexibility to do what I want with my own network.
Why do I have multiple computers? Playing with different operating systems etc. isn't really the reason as virtualisation is pretty good these days. It actually comes down to locality of the machines. I want my hard disks to be in a cupboard as they are quite noisy, but I want my screens and keyboard to be on a desk. So I've got a NAS in a cupboard and a (quiet) PC at my desk, plus a (silent) media centre in my living room and other stuff.
One thing I would say is don't be tempted by rack mounted server gear. Yeah, it looks cool and you can get it second hand for not much, but this isn't suitable for your home. Use desktop PC cases with big fans instead. Rack mounted network gear is pretty cool, though.
That depends on the server. Super Micro makes some rackmount servers that are shallow enough for a network cabinet/wall-mount rack and fairly low power. They also have a real OOB mgmt system, which can be really helpful. https://www.supermicro.com/en/products/system/iot/1u/sys-510...
I agree that typical deep-rackmount servers are more trouble than they are worth for a home lab.
You can find regular-PC-case-shaped motherboards that have real IPMI management baked in, but you do have to be willing to pay more for it, as they're usually (always?) found in "workstation" boards.
However, you do need to do a bit of research to make sure the advertising claims match up to reality. Some of these "IPMI" ports require custom Windows-only software and don't let you remotely control much at all.
But unless you have a real rack it's gonna be a bit of a pain, because it WILL end up being stacked on other pieces of gear in a rackish tower that requires full dismantling to get to the piece you want to work on.
If you go off-lease rack equipment, go whole hog and get a rack and rails too. Rails can be a bit pricey, check the listings for those that include them - rails often work for much longer than the servers so the big companies that liquidate off-lease equipment don't include them; smaller sellers often do.
Try to find one of these hundreds of small form factor products that has anything faster than a 2.5Gbit nic and despair! What good are these machines if you cant even get data in and out of them?!
The list of hardware with 2x10GBe or better connectivity is amazingly short:
- Minisforum MS-01
- GoWin RS86S*, GW-BS-1U-*
- Quotom Q20331G\*
- Mac Mini (1x10gbaseT option, ARM, )
- HPE Z2 Mini\* (1x10GbaseT option)
Minimum of 2 NVMe
Minimum of 2 additional storage devices (SATA or more NVMe)
Minimum of 2 10GbE
Dedicated or shared IMPI/OOBM
Minimum of 8 Core x86 CPU
Minimum of 64GB RAM
Looks like the X10SDV-TLN4F[1] has everything you mentioned besides a second m.2 slot. Although its 8 cores they are a few generations old and low power (45w TDP on x86). For true compute its not exactly fast but for something like a high performance file server in 1U connected to a disk shelf, they are really nice.
[1] https://www.supermicro.com/en/products/motherboard/x10sdv-tl...
On one hand they are able to still sell them because nobody is competing, but what is the reason for the lack of competition?
I have a theory about Broadcom buying and blocking up the low end PCIe switch market to purposely hold back this segment, but I'm not sure how significant that is.
I made the mistake of getting Seagate 10TB rock tumblers, but they’re the only source of real noise. WD would probably have been quieter.
I have this and my switches and a ham radio base station all racked up on top of the cabinets in my laundry/utility room, out of the way but not hidden away, so I can still notice if the fans are too loud or the wrong lights are blinking or not blinking or there’s weird smells etc.
If you don't need it built-in, external adapters are a good way to get to 10GbE and even 25GbE.
https://www.qnap.com/en-us/product/qna-t310g1s
https://www.sonnettech.com/product/thunderbolt/ethernet-netw...
I'm not opposed to Thunderbolt though; let me know when someone ships a 8+ port thunderbolt interdomain/multi-root PCIe switch. That would be a good solution.
What's the second link for? Just more bandwidth, or is there some crucial use case I'm missing?
Of course you can still build routers and everything with a single interface (which is why I noted the single port hardware in the first place), but if you are going to design something specific to the mini-server market, the cost difference between a single port and dual port controller is negligible enough that there's no reason not to include 2 or even 4 ports. The GoWin machines simply toss in an OCP nic slot and call it a day. That would be a fantastic solution to see adopted by other vendors.
I once moved a server from one room to another by moving its redundant power cables one at a time to a UPS on a cart, then swapping its short network cables for longer ones one a time, wheeling the thing to the other room, then “reversing” the procedure in the new location. Zero downtime!
Zero point really, but fun in the way I reckon a home lab is meant to be.
This is the way. HP, Lenovo, and Dell are all making identical small form factor PCs that are more than enough for most people at home. They are also very silent and power efficient.
I have a Lenovo M720 Tiny and a Dell Optiplex 3080 Micro (they are virtually almost the same). You can change parts, there are ample ports available and you can pretty much run any OS you want.
I think I paid less than $130 including shipping and sometimes it (anecdotally) has better performance than my home PC which is a full tower I bought new 4-5 years back.
This deserves a shout out. These things are everywhere on eBay and they Ryzen ones rock.
Was hoping this article would delve into hosting your own email, websites and services and the steps to expose them to the public internet (as well as all the requisite security considerations needed for such things).
A homelab is just a normal PC unless you're doing homelab stuff.
I've tried on numerous occasions to host my own services at home and use dynamic DNS to map to an IP, but when things don't work, it's hard to know if it's the hardware (consumer modems, routers, etc.), dirty ISPs who want to force you into a business tier to open up anything, some combination of the two, or something else altogether. Are the problems so one-off and specific that they all require bespoke problem-solving?
I would love it if anyone could share foolproof resources on the subject.
As for your actual problem, a lot of ports are blocked by ISPs because a LOT of computer worms have used them to spread malware in the past. However you usually can test to see if the port is open as well as try alternative uncommon ports like 47364 to see if they work.
Some ISPs, although I believe mostly mobile ones, also use something called CGNAT where your Internet IP is actually an internal network IP within a part of the ISP, and you can usually test for this by checking if your IP is in a private range.
All of this is very interesting, but not very useful. I'm personally most intrigued by how people are maxing out their old hardware for years, like many examples in this thread
It's fairly common for people that like to muck around with this stuff (or learn it for whatever purpose). Think of it like a hobby I guess. :)
Many people get so struck by gearlust they lose sight of what piqued their interest in the first place.
I'm not immune, I don't think I'd have a Unifi setup at home if I hadn't seen the flashing lights and convinced myself I needed some.
The nice thing about off-lease enterprise hardware is there is so much of it, and it's enterprise. You can easily find replacement parts, and working on that stuff is a joy, everything is easily replaceable with no tools.
My biggest (current) problem is finding something that supports a ton of SATA drives and still is low power (though at some point even the power usage of spinning rust can become noticeable).
I live in the icy wilderness, so excess power to heat isn't a problem for most of the year.
My guess is that if I was used to a much faster machine, then the machine's slowness would bother me, but it in fact does not bother me. (The machine has an NVMe interface, and I have a fast SSD in there. I have the N100 CPU because I couldn't find the N200 or the i3-N305 for immediate delivery from a non-Chinese manufacturer.)
"Be aware, the video server and sync server will be offline between the hours of 8PM and 10PM on Friday, March 8th for hardware upgrades. Expected downtime should not be more than 20 minutes but please plan around this disruption to essential services. Other services should not be affected during this work."
Like someone else mentioned in this thread: It was really best described as a "home network with more than one computer." But you can't really put that on your C.V. under "experience."
"Improved SLA uptime on client's network to five nines. Upgraded storage controllers in a live environment without disruption to established services. Deployed k8s cluster into a production environment serving thousands of hours of compressed video files with real-time format transcoding."
I will freely admit, my home network probably needs to be pared down a bit. The problem is the salvaged hardware and good deals at the liquidation auctions accumulates faster than I can cart it off to the street corner. There's also far too many laptops floating around the house doing nothing useful that could be donated away. A lot of the homelab is salvage or bought at (in-person) liquidation auctions. There's some new stuff, but I rarely need the latest and greatest for my setup. In my home network diagram I try to document everything that touches the network so I can stay on top of the security patches, and what has access to the outside world and in what way.
https://justinlloyd.li/wp-content/uploads/2023/11/home-netwo...
> Pros: Less noise, easier cable runs.
> Cons: can get hot depending on where you live, roof leaks, humidity/condensation, and creepy at night.
Where do attics not get hot? I live in northern climate, and ours regularly gets hotter than 140F during sunny summer days.
And I say hotter, because the remote temp sensor I have up there maxes out at 140F. I wouldn't be surprised if it actually gets up to 150F or more.
It's nice because that loose, shredded insulation is garbage and so easy to mess up (walking on it can change it from R40 to R20, for example). With a conditioned attic you use standard batting in between the roof members -- impossible to mess up.
No reason to go crazy with your home server if it is only you using it. You can get away with a crappy laptop that your roommate is done with.
How much does it average in power draw (watts) from the wall?
It allows you to go deep into settings and configure anything to your heart's content. But, if you want to ignore it for a year and not mess with it, it will run as smoothly as an Eero or a similar consumer product like that.
I have a Purple model[3]. I initially balked at the price, but it performs well, and there hasn't been a single day in the past couple years where I've regretted it or looked at something else.
It supports just about any router feature. Has robust and easy to configure device/family profiles. The quality of the Firewalla software and the mobile app is excellent.
---
1. I manage a quite large WAN during the day. I don't want to do it at night, too.
The software is open source apparently? https://github.com/firewalla/firewalla Anyone running this on their own hardware? Or in a VM homelab :) ?
Nearly everyone has a spare PC lying around collecting dust, and within an hour it can be running a dozen VMs . Reclaimed HP & Dells are cheap and include a Windows Pro license. otherwise they can be had for $20
I recommend Alpine for the guest OS. It's fast to set up, natively supports Hyper-V hosts and compact (about 1/10th the size of a debian guest).
https://learn.microsoft.com/en-us/windows-server/virtualizat...
If I reboot windows, my Debian VM comes right back up, with all the docker-compose services running (a la `restart: always`). Like nothing ever happened.
So I just don't think this is a problem. But that's just my experience...
fwiw my linux machines get periodic kernel & firmware updates that also need restarting
HyperV is perfect for what I want to do, run a dozen or two sandbox and experimental VMs. I can pass through hardware, assign raw volumes, partition out network interfaces for routers and other shenanigans and it's stable too. And with RDP, I just use the mac version of the remote desktop client to use it interactively, which also works surprisingly well even over tailscale on another continent!
Use what you got!
Remote Desktop is really good, too. I use it all the time from my MacBook (though I mostly SSH, naturally). WinSCP + midnight commander are all the extras I need.
If you have enough memory/CPU on your workstation/gaming machine, Just 1 PC works great.
Hyper-V is very limited in hardware support, you won't be able to pass-through stuff. And you always pay for VM layer, while in Proxmox you could use containers which give almost the host performance.
https://learn.microsoft.com/en-us/windows-server/virtualizat...
https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-...
Hyper V Server 2019 is also free you don’t have to pay for anything https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-...
And you can install a full GUI too https://gist.github.com/bp2008/922b326bf30222b51da08146746c7...
For networking I can recommend Fujitsu Futro S920 thin clients, especially for EU users. They are dirt cheap here, and with aftermarket riser & intel NIC they are great routers with opnsense/pfsense. Apart from that, I have a couple of raspberry pi's running dedicated tasks, and space heaters (e-waste tier enterprise desktops) running BOINC during coldest months.
There are just SO MANY perfectly good computers that are bound to be e-waste that would be good for this.
Old laptops / desktops can be better - especially since you can get newer/more power efficient units without all the noise. But I started moving to even smaller SBC based stuff (0)
* Hardware: A regular but small (SFF) computer + Synology NAS
* Software: Ubuntu distro + Docker containers bare
* Configuration: Git-versioned Dockerfiles and docker-compose.ymls
* Storage: The NAS with Synology Hybrid RAID
* Backup: TBD... haven't figured out a cheap and automatic way to backup 20 TB of data off-site
I run cameras, a media server shared with friends, a shared place to dump project files, a DNS server (just in case the Internet goes down so that everything local still works) and supporting services.
The best I've come up with so far is segmentation via VLANs, then adding many rules to deal with inter-VLAN reachability (e.g. should it be forced to transit to the firewall and back? Can my laptop reach out to the IoT devices, even if I block connections in the reverse direction?).
Beyond this, I'm not sure how to best handle similar segmentation on my few servers (all running proxmox), e.g. whether to expose VLANs to the VMs (which I kinda need to do for the pfsense firewall, maybe?).
The biggest thing is how to keep these network configs in sync, e.g. when adding a new VLAN, etc. I basically have an Org file that I write all this down, but I'm guessing there's a better way.
Also, if anyone wants a cheaper alternative to Ubiquiti, Mikrotik has served me well for years. I'm considering switching to Ubiquiti stuff though, as they seem to have a solution to synchronize state between multiple switches that is much better (vs. manually using Winbox on each individual router).
The quality of software out there these days is insane -- there are projects that cover so many crucial functions (tracking expenses, playing music, managing data, etc), that it's becoming really hard to track them all.
I've been working on converting the site to a directory and it's not quite there yet -- but some people do like the massive wall of projects.
I have a beginner setup: a single NUC and a cheap NAS with NFS.
I have yet to come across something that this setup cannot handle.
When I saw the picture of the server mounted in a cardboard box, my first thought was about fire hazards. It turns out that might not be as big an issue as I initially thought, since paper won't spontaneously catch fire until reaching ~200C [1], and the operating temperatures of computers generally shouldn't exceed 100C. However, I'd still be a little wary about a cardboard setup, considering the potential for electrical faults, the heat generated by power supplies and the risk of inadequate ventilation leading to hotspots. These factors could present a significant fire risk. My initial judgment of the setup may have been too quick but not entirely unfounded I guess!
Started with
- PiHole to cut down on ads and try to keep the kids from clicking malware links
- added network attached storage to make it easier to share files with the family
- added Jellyfin to share music with people in the house (kids gotta grow up listening to good music)
- added syncthing to be able to ditch dropbox and still keep multiple computer file folders synced
- ran a Minecraft server, then a Terraria server, then shut those down since we moved on to other games
- turned on an ntp server since ddwrt router wasn't getting the time from the internet reliably
- added librespeed and ittools for quick wifi-speed-testing and to convert docker-run commands to docker-compose commands
- needed a webpage with links to start keeping track of all of this, so lighttpd it is...
- smokeping and blip were fired up when the internet got flaky and I needed more data to explain to the ISP what I was seeing
- Got a 3D printer, and Octoprint made it easy to start and monitor prints without having to pull a microSD card out of the socket every time.
- Figured, while we're here, that I might as well explore the smart-home thing with HomeAssistant -- now we have a handful of lightbulbs and smart power switches that mean you don't have to get up to change the lights quite as often (not everything is on a smart control, just a few out-of-the-way things. Oh, yeah, handy during Christmas-time to turn the tree lights on and off without having to fumble behind the tree for the little switch.)
- That, of course, lead to an MQTT broker (mosquitto), which is kind of a budget pub-sub broker
- And now that I have docker logs with things I need to debug, Dozzle serves those logs in a handy webpage. Not secure, but it's not like this is enterprise server.
- Then I wanted to play around with timeseries data (just to learn), so (running out of RAM at this point, so I couldn't just fire up Grafana) kept digging until I found VictoriaMetrics, which was quite happy to catch metrics (fired at it from Benthos) and visualize some 3 years of hourly Beijing weather data. (Seriously, I'm still amazed how lightweight, simple, and easy to use VictoriaMetrics is)
- currently thinking of kicking the tires on the NATS message broker and see if I can get the multi-node clustering to work. No real purpose other than see what it's like and dream about silly developer projects.
- also thinking of exploring SeaweedFS -- might be nice to have horizontally scalable storage, as each of my nodes are scattered around the house depending on where I can put an Ethernet drop.
- after a few years going along this journey, it has the perk of having helped me get my current job, which is pretty cool.
So... it's mostly a plaything, a sandbox, but a handful of things run "in production" at home and make life around the house slightly more convenient and less risky. By being choosy with the software and a bit aggressive on the container RAM limits, I've got 27 containers running on a Raspberry Pi. Hey, gotta take my bragging rights somewhere.
I have mostly consumer (non-rack) equipment in a small Ikea closet. Looks less nerdy, which also improves the WAF.
SBC based stuff is plenty powerful for most stuff now, MUCH cheaper then Nucs, and SILENT! (Great for my sanity)
I can hear my racks but they're about as quiet as the furnace until I do a really big compile.
On reboot they go into full speed fan mode until the controller comes online, that sounds like a jet taking off.
These are mostly 1st-gen i7-4600U Haswell's, upgraded to 16GB RAM and 128G+ SSDs. I ripped them out of their cases, added heatsinks to the new SSDs, and then "racked" them by wedging the front-bottom mainboard corner (if plugs are oriented to the rear and the heaviest plug, AC, at the rear-bottom for balance) into a slotted piece of super-strong and right-angled cardboard that came from inside some re-inforced shipping box (I think it was an X-mas tree).
I assembled all of this juuuuuuust before COVID while learning K8s and craving the physical plan aspect, as I had just transitioned to a remote SaaS role from one that involved multiple physical DCs and travel.
My "go small lab" initiative turned into some sort of Tribble fable.
For /etc and my source code repositories, I use restic and cloud storage, because this data is small (< 1 GB). (Also, I'm on rural internet, which is metered and slow. So I can't use cloud backup for the big stuff.)
The whole system is not quite 3/2/1, but it's close enough for me.
Why are so many different components needed?
If you want something for practical purposes, /r/selfhosted actually has you covered. It’s usually jellyfin for movies, etc. And you use a NAS or a PC, but the idea is normally that you setup something reliable and don’t lose your data, and it does not require too much tinkering. Maybe once in a while run some docker container to test it out. what I have now.
Then there are people who get many raspberri Pi’s or NUCs and basically play ‘build a datacenter at home’ game. It’s good for tinkering and you often mess it up and start from scratch. I’ve done that, it’s kind of fun, cool experience, learned a lot about networking, installed Kubernetes, then sold the machines.
Then a third group are people who build a whole data centre at home AND run serious applications on it. These guys need to buy themselves a motorbike or something.
I don't think the insult to a generalized group of people is necessary.
Some enthusiasts have overkill racks, 40Gbps fiber, enterprise-class switches and vlans, etc, etc ... because it's their enjoyable hobby. The over-the-top components and making it all work is all part of their fun.
It's not a hobby I'd personally pursue but I do understand why it can attract a passionate set of people.
I find it highly convenient personally, I have a data cap so moving large stuff files around happens over 10gbe locally instead of over the internet to aws or whatever. It's also all mine so from a security and privacy perspective it has a lot of advantages.
* calendar/contact sync (webdav)
* personal crm
* notes webapp
* email (yes, my primary one. helps when you're running business internet from home rather than a random ip on a vps)
* mediawiki for writing
* rss reader webapp
* internet file sharing (basically selfhosted dropbox)
* virtual tabletop for d&d-like games
* reverse proxy/dns/cert management (a let's encrypt post renew hook that scps the certs where they need to go)/zfs
This runs on an old rackmount server under a table using KVM/QEMU (once you get the commands written it your notes it's not so bad to manage purely from the command line), a cheap managed switch, and x86 router. I back up important stuff (email on the server, then lots of stuff on my desktop) to BackBlaze
Looks like the author of the story is also dabbling into reliability engineering as well; having hardware can be a little helpful for that too since there's often vendor quirks as well.
1. Enjoyment of building an enthusiast system from select components,
2. Having a more capable and flexible system,
3. Learning how networking works by being your own network admin.
You don't need all the components. Many will get more just because they want to (see point 1), but mostly you get them because you need to (see point 2).
- AdGuard / PiHole - Plex server (plus the whole suite of *arr -- I gave up Netflix during the password sharing debacle) - Home Assistant (I'm on a huge data-ownership crusade this year) - Stats tracking for my car (TeslaMate -- this is such an amazing piece of software cannot recommend it enough if you've got a Tesla) - general tinkering
I'm currently building a house as well and I plan on deploying a way more complicated networking setup (full-ubiquiti) so I can get rid of all my internet-connected security devices (cams and doorbell for example).
Honestly... I used to build PCs (I still do when I need one) and now this fills that gap for me. I spend way too much money on it and it makes me happy 'cause I'm having fun. I get to tinker and break things in the privacy of my home. I get to learn about Linux administration (this is honestly the least fun part) and networking.
Usually self-hosting things and making things more complicated for fun.
> Why are so many different components needed?
Learning experience. Many home lab enthusiasts are sysadmins(-adjacent) and use it as a learning experience.
* Streaming media server with Plex
* Jenkins build server
* Game servers (Minecraft, Factorio, etc.)
* Photo syncing with Immich
* File syncing with Syncthing
* Alternative frontends for Reddit and YouTube with Teddit and Invidious
* Home automation with Home Assistant
I set up a Mellanox mesh network between my three compute nodes a couple of weeks ago. 56 Gbps Infiniband is pretty sweet, although my CPU is too slow to fully take advantage of it (I don’t have RoCE working yet, so it’s IPoIB). Still, a ~20x speed up for my Ceph cluster on NVMe is lovely, and makes it actually viable for use.
For example - Proxmox, XCP-ng, VMWare ESXi, Kubernetes, docker, etc, etc. All this require knowledge and experience.
It will be a bit general but what's the best use case here?
I mean try to build a home server, self host some services but there are so many options.
- Proxmox, run a VM for each service. And it supports LXC too.
- A vanilla Linux server which runs a type 2 hypervisor (QEMU) for each service
- Docker can be also done either in a single server but some people do it with something like Unraid (which is for NAS but support virtualization too). Very similarly you can install Docker in OpenMediaVault too.
Feels like there are so many ways to achive the same thing
You can start self hosting on a mini PC if you want.
This is the biggest thing for me, I would love to have a big fancy lab setup at home, but when you start adding up the power consumption and put a monthly dollar amount to it, then it becomes less exciting. Especially if you live somewhere like California where home electricity bills are incredibly high nowadays, it is extra painful.
There are some low-power setups that are great, but it seems like once you start using rack-mounted equipment, the machines get more power-hungry.
TF to bootstrap vms + rke cluster. TF to deploy helm charts. Still some per app config/backups to mange but I just copy container volumes to all the machines in the cluster with the csi plugin and then back everything up to zfs nas via truenas on NFS.
Runs primarily on 3 dell rx30 series, and then my desktop and htpc are also in vsphere running gpu centric things when I'm not gaming with them.
Vmug is pretty perfect for this -> $180/yr = 16 or 36 physcial sockets.
[1] https://us.ecoflow.com/products/river-2-pro-portable-power-s...
I considered moving them to the garage and then running USB-over-fiber to access them but have a ton of flight sim gear I need powered as well.
Here's my naive question: What is the point of a home lab, i.e. what makes it better than running stuff on the machine you're developing? I can see some advantages in networking applications (as mentioned in the article), but I have the impression this is not too critical for most applications and devs.
Don't get me wrong, I'm a fan of building stuff like this, and will always be able to find an excuse to do it anyway. :)
Its murky because there is a bit of backstory. There used to be a homeserver subreddit. The people running commercial heavy rack servers got fed up with the peasants and created homelab sub. Then overall quality & vibe in homeserver sub dipped so everyone migrated over to homelab. So now you've got everything under the sun that is at home and switched on 24/7 being called a "homelab". Leading understandably to people being confused about the term
>What is the point of a home lab
People do it for different reasons - "own your own data", learning (IaC/docker/kubernetes/networking). It's also quite empowering for experimentation. Most of these setups are running hypervisors. So creating a VM is a 20 second affair. That sort of blank slate is useful to me for dev work. Local gitlab server, local CI runners, local kubernetes cluster, postgres server. It's all there, free, fast, accessible and ignoring initial layout effectively free.
In contrast, your dev machine may need to be rebooted any time, or switched off if you leave for the weekend.
A second defining characteristic of a homelab is that part of the rationale for owning it is in order to tinker with it, to try out new software packages, to play with tools & concepts (VMs, containers, admin and remote admin tools, KVMs).
Ultimately, you may define a homelab as a "physical server setup for learning and for running personal services". Homelab operators may also take pride of the independence from large IT or cloud providers attained by operating their lab.
The most important point is to be available all the time in multiple devices that you own, that's why some folks don't want to do these things on their main machines. A laptop you could take it away, and your family at home would/could lose access to the things you're hosting. A beefy desktop might be a bit too power hungry to be on all the time.
It’s a place to learn while playing pretend. LARPing enterprise IT if you like. Why wonder what the point is?
https://www.reddit.com/r/homelab/
https://www.reddit.com/r/minilab/
My "homelab" for now is a single Raspberry Pi (I want it silent and not take much power when idling). But from there, I can power-on my desktop, via wakeonlan.
I have a nice used Dell Poweredge T320 with a single Xeon E5-2428L that's been running on only an SSD (and then a raid 1 of 3.5" HDDs) for over 8 years, but electricity prices are rising and I wish I could measure it precisely, idle and under load.
And even without external tools, historical power usage logs are available via the APC Web UI.
While I don't currently log anything externally, I use an xbar[1] script[2] to display UPS output current in my Mac menu bar.
Has anyone ever needed CAT7 (or better)? CAT6 works for 10 GBit at a short distance, but in my experience (in corporate IT) I've literally never seen a 10 GBit Ethernet port! SFPs are the shit, direct attach SFPs are cheap as chips and fiber ones aren't even that expensive anymore. And of course fiber will handily beat twisted pair in signal integrity at any distance.
I have one of those SoC Intel boxes with 4x2.5Gb LAN running proxmox, running OPNsense for my router. I had to get a fan to keep it from overheating.
Proxmox is a great SDN/VM host and I haven't fully explored monitoring and automating it.
Device is finicky with sata adapters. I’m using some Marvell controllers at the moment.
https://www.klennet.com/notes/2022-11-25-zfs-and-ecc-rant.as... is a pretty good overview. ECC is "nice to have if you can get it cheap enough" but it is not a ride or die.
Some starter goals:
* Get a Raspberry Pi
* Assemble the case for it, format the SD card, start the OS
* Install PiHole & Tailscale
* Setup Tailscale on your phone so that you can use PiHole on the go
* Adjust your router so that most devices go through PiHole
* Get a more powerful router or OpenWRT software for the router so that you can do hairpin NAT to reroute everything to the PiHole.
* Attach a hard drive to your Raspberry Pi...
I chose Nextcloud and Jellyfin when I started in 2020. Before then I was running a few random things like Syncthing and Samba mostly just for myself. In 2020 I decided to try and apply a more thoughtful, disciplined process. Good security, reliable backups, useful for my whole family, etc.
Shameless self-promotion: https://selfhostbook.com . My book covers justification for self-hosting, fundamental skills, and homelab (or perhaps "homeprod") setup with Ubuntu, Ansible, Docker, and Traefik.
I wrote it for folks with some light sysadmin/programming skills. It's a good general starting starting point for self-hosting.
Like with learning a programming language, having a specific goal helps a lot.
Reading other people's posts about their hardware, how they use it, and what problems they solved with their homelab can be a source of inspiration that sets you on the right path.
Like others on this thread have said, while it's tempting to immediately dive into racks, patch cables and network diagrams, you can achieve A LOT with something simple like a small NAS or a NUC. Even a spare laptop that you can repurpose might be more than sufficient.
Unless you're trying to simulate managing enterprise hardware (and I mean physically managing it), the greatest value lies in learning the software side of thing, IMHO.
Reminds me of all the "so simple your <older female relative> could use it" comments on older tech product advertising.
Just smells of yet more boys club humour that adds to the difficulty of including women in the hobby + industry. It's the small jokes that we're supposed to just quietly smile along with, as well as the big, that cause problems.
Otherwise, nice post! I've had my homelab for years now and have been looking at expanding, definitely some things there I'll be looking at.
Anyways to your original point I feel as if online people feel like marriage is a bunch of negative compromises that are always one sided or something. Homelabs are great but please don’t put loud servers and PC’s in the living room. :)
I would would have have no problem reading an article which makes jokes about the husband being driven to insanity by home server, I would have appreciated the diversity of perspective.
Some have suggested some truly cumbersome terminology to replace it which is unfortunate because you very quickly run into the same problem the vegans have: If your choices are too unpalatable then reasonable people (who don't care or share your values 100%) will avoid them.
You need to make the choice easier. (hence why a lot of Vegan focus right now is on having enjoyable alternative meals).
[0]: https://en.wikipedia.org/wiki/Wife_acceptance_factor
EDIT: I'm not going to argue this because if you're reading my comment you've already decided whether you agree or not and you will not be productive to talk to.
I will answer some common suggestions and why they're not reasonably sufficient and you can argue amongst yourselves about it. I'm not a sociologist or psychologist but I am human so take it with that lens.
* Partner as a replacement for "wife" is unfortunately more syllables making it just a smidge more annoying to speak, so people won't; but it's the closest to what is intended and mentally equivalent for most people. As a bonus it even encompasses the unmarried! Partner is also (commonly) said lovingly so it doesn't create distance in the mind of the speaker.
* Spouse is an uncommon enough word that it doesn't come to mind quickly enough; it also unfortunately comes across as unfamiliar (IE: not from a place of warm affection) and clinical.
A good replacement term should be something endearing because it's important to point out that this isn't a term of derision, the true meaning behind the phrase is that they love their partner and want them to be happy. It's just a boomer-humour type joke to hide behind "because I dont want a divorce" or "I fear my wife", which is part of the supposed humour -- so it needs to feel familial and affectionate.
I don't think it has anything to do with genders per se but is an old adage with any hobby.The historical implications do point to a bias in understanding of 'roles' but I'm more comfortable with the approach that instead of rephrasing old adages we become comfortable with the idea that our understanding of the specific adage has changed.
I think not changing the adage and instead focusing on the shift of understanding can also point to how much we grow as a society, and we can relish in our evolution of understanding as it pertains to historical beliefs.