Home lab beginners guide (2021)
haydenjames.io
haydenjames.io
I feel like we live in a world in which it's either racks or cheap VPSs. In reality, at home, we have some serious CPU horsepower just spinning billions of empty cycles per second. Consumer hardware is insane.
I've handled 10's of thousands of unique visitors per minute and more than a couple front page of Reddit + Hacker News herds on this little laptop through a residential ISP.
Here's my setup (pic down at the bottom): https://kiwiziti.com/~matt/wireguard/
Depends, I get a lot of utility from mine, as it manages my media collection for streaming at home and on the go. I've tried using SaaS alternatives and managed hosting of the apps I run, but those experiences were both lackluster and relatively expensive. And since the apps I run and my media collection aren't locked behind proprietary systems or limited APIs that might disappear, the amount of integration and automation makes for a very pleasant experience.
> In reality, at home, we have some serious CPU horsepower just spinning billions of empty cycles per second. Consumer hardware is insane.
I just add my old devices to my cluster and call it a day. Even ancient hardware is suitable for it, especially if you're using old laptops that are optimized for power savings. Even old Core 2 processors in laptops can idle at a low wattage, and TDP can be less than a light bulb's when maxed out.
One of the most irritating things about "homelabs" is that most people seem to think a "homelab" means "a rack of very expensive, way-overspec'd ubiquiti gear, an OTS NAS unit, and a docker container server running media server/torrent shit."
I have a laptop running a dozen different containers - bookstack, torrent client, rss reader, and so on. I don't think of it as a "homelab."
Some people are interested in only one of these, so the distinction is not pointless.
Distinction springs from the person themselves.
I think ClumsyPilot's way to put it was just fine.
I just said that the distinction is not "pointless" or "gatekeeping", as some people accused, but pretty common convention which can be quite useful for readers. In particular, you are pushing back against something nobody said.
My "media server" (browser and downloaded media played via smplayer on a stock Ubuntu install) emerged from an experimental server running a lightweight distro that I used to do anything and everything from. Once I found that which parts of the media usecase fit into my partner and my lifestyle, it graduated to a stable decently-specced Ubuntu machine that is rarely touched other than for updates and downloading new content.
It looks to me that rolling your own media center is something to experiment with and learn something from.
Not to mention even if it was just a Plex server, that still meets your criteria.
It grew over time to serve many purposes, has seen many stopped/failed attempts, has had many lives (I mean, recreate from scratch to learn X, Y or Z).
There's not a particular day or a particular addition or a particular level of complexity when it became a homelab, I think of it as such from day 1.
Here are a few of the deployments that originated in my home lab but ended up in a prod environment at work
1. Replacing hardware load balancers with haproxy. (This started with a few options, including Nginx, and some others. But haproxy and it’s web management, csv monitoring gave me the best capability to integrate at work)
2. Vpn appliances for covid. I was able to whip up 2-4 scalable VPN appliances based on openvpn in “1/2 a day” at work because I was able to flesh out most of that at home.
3. Vulnerability scanners
4. HIDS security tools. In the end we went with a OTS vendor but options like ossec, wazuh etc were ruled out in a lab.
5. Ansible (over some of the other options)
6. Squid for reporting on the HIDS mentioned above.
There’s probably more. I know there is. And point blank a lot of this stuff had mock ups done at home because I have full control and am not subject to auditors etc when evaluating them. Whereas I do thst at work and I have to do more work writing up justification or change requests etc. it’s just easier at home.
All that said I try to keep the house as flat and plain Jane as I can.
I also have a lab hypervisor (Dell T30). This is where I feel free to experiment with VMs and accidentally on purpose have to rebuild it every once in a while, take it down to swap out hardware, etc.
My "media server" consists of a web application, backend application, multiple *arr services, transcoding automations, fibre termination, user account management shared across multiple machines and services, multiple VLANs and LUNs, etc.
All these are spready across 16RU or so, but really only serve as a "media server".
With less technical management, I've had repeated, and bewildered, conversations trying to get them to understand that our one "computer" sitting on my desk is many many times faster than the "server" our IT team provides. "But it's a server!".
Many times faster doesn't mean shit if it can't be remotely lights-out managed and its hardware monitored using standardized tools (or at all.)
Many times faster doesn't mean shit if it doesn't have redundant PSUs.
Many times faster doesn't mean shit if failed drives can't be hotswapped.
Also, the computer sitting on your desk is not "many many times faster" than a current, or even few years old, server.
Etc.
If you want better hardware from IT, tell management to give them more money. IT is almost always viewed as a cost center and given a shoestring budget yet asked to do, and be responsible for, he world.
You know how you're experienced from all your years as a programmer? Imagine IT people are the same, instead of assuming they're all idiots who are too stupid to go out and buy desktop computers instead of servers like your genius self.
The server is a big pie. If you're buying a single slice, then yes, it's very very easy for a cheap old desktop to be way faster than a cheap VPS.
> Imagine IT people are the same, instead of assuming they're all idiots who are too stupid to go out and buy desktop computers instead of servers like your genius self.
It's the managers that are idiots. Not everything needs to run in a datacenter. Some things really are kittens and not cattle.
Unless you're hoping to monetize that spot on your desk, the real estate market means nothing in terms of cost.
> Many times faster doesn't mean shit if it can't be remotely lights-out managed and its hardware monitored using standardized tools (or at all.)
What stops you from "using standardized tools" on a box you own?
> Many times faster doesn't mean shit if it doesn't have redundant PSUs.
What leads you to believe that all those 9s are relevant or even not comparable with cloud alternatives? In fact, I'm not sure that the latest rounds of outages at AWS allow it to claim more than 3 9s during the past year.
> Also, the computer sitting on your desk is not "many many times faster" than a current, or even few years old, server.
Actually, it is.
But seriously, they were comparing to the server they got, not the one you have or can provide.
It’s entirely reasonable for the IT team to provide a VPS that doesn’t have nearly the amount of power for a application that’s barely used. Doesn’t mean it’s easy to explain to management.
Nearly all of your assumption here are incorrect or flawed, except the redundant PSU (we only had one). But, I do think they're just like me: working in a non-ideal environment with constraints outside of our direct control. The non-ideal constraint that they had, in that instance, at that time, was that they could only give us a VPS with 4 threads. It wasn't possible to do what we needed with their server. Or, to put it into your language, five nines doesn't mean shit if, in practicality, it makes a reliable space heater.
I think a lot of people build a homelab to learn about technologies and get realworld experience deploying them. That was what drove mine for a couple of years. Once you’ve mastered servers and networking and so on, then it just becomes a fun hobby, I agree there. But someone who wants to get into networking and the like (and who lacks experience) is definitely going to need to practice with real or simulated networks to get good at it.
And yes I was around before the “cloud” was a thing. I first networked a Mac LC II and PowerMac 6100/60 to play with a Gopher server.
Wireguard is both very frustrating and very cool. I'm currently using it similarly to give a VM a public IP, and i'm testing the details on getting multiple SSL/https hosts behind that single IP, which is something you couldn't easily do a decade ago without the host with the single IP having all of the certificates and "MITM" the entire session.
Speaking of "CPU horsepower" i just replaced a 1.5kW HP[0] server with a .2kW Ryzen 5950x "server" that is about 5% faster overall - don't forget that old stuff, while capable, adds to the electric bill, usually at a constant rate.
[0] the iLO (lights out) actually reported the server's power usage in BTU. It drew the same power as a portable space heater.
What I’ve observed is people on subs like r/homelab and r/sysadmin ridicule people who appreciate the available horsepower with modern consumer tech because “no ecc memory” or the like and I wonder if you people who are looking to make labs using the latest ryzen or i7/i9 (really, I’m thinking of getting started by converting an old thinkcentre with a 4th gen i5, possibly undervolting the cpu, and 24gb ddr3 into a pfsense router and some sort of server) will really be missing out on some necessary enterprise feature?
Old HPE servers have jumped in price though. Last year you could buy insanely powerful stuff for under $200 but it's all $400 and up for the same gear at the moment.
ECC is about long term stability and data integrity. For a router, meh, the network protocols will deal with any flips. For a file server or database it's better if those random bit flips don't happen to critical data.
AMD based systems can sometimes be forced to use ECC mode even if the BOIS doesn't support it.
ECC is more important in systems with very large ram footprints because there's that much larger of a memory footprint area for cosmic rays to corrupt. If you've got one-two sticks of ram, and you're not running vital business data, meh, it's not required.
I really like ECC. But I'm not really willing to pay a significant premium for it.
I run companion systems to production out of my house, mostly development lanes comparable to production deployments. If they're down, it's really not mission critical. I also run my home security/surveillance systems. The other significant systems are those related to my children's computer lab.
One of my first real homelab setups, was 6 raspberry pi’s, laid out as two load balancer, two web app servers, and two database servers. The “unreliability” of pi’s running everything off SD cards was a _feature_ because it gave me lots of experience in how my (work) software stack running on our typical AWS design held up in face of failures.
It is not validated. So YMMV, but it works and is seen by Linux for me at least .
Yeah. I've got my search engine basically a beefy gaming PC with no graphics card and 128 Gb ram. I've not only gotten supposed HN death hugs multiple times, I've had my search engine see multiple queries per second without as much as flinching. It took Elon Musk tweeting a link to one of my blog entries before I started getting more traffic than my computer could handle and it started dropping a few connections.
Modern consumer PCs are ridiculously powerful if you make good use of the hardware.
Are you hosted via a residential ISP? It's my hunch that peering agreements favor routes of consumer -> data center -> consumer as opposed to consumer -> consumer. That's mainly why I tunnel. Has that been your experience?
I was hit by a botnet after my first big HN hug, so right now at least the search engine goes visitor->cloudflare->server, but if anything that's just added a bunch of ping.
I'm also doing crawling and so on on the same network and it's really not bad for that either. Granted my crawls are fairly slow and very far from saturating any connections.
To be fair they're not spinning every single piece of the CPU.
My desktop can play games from 2009-ish, but at idle it clocks down to like 30 watts.
It could play games from 2011 if I put the GPU back in, but the GPU's idle power draw is ridiculous...
Seems too narrow to me.
I am not really interested in reading about this kind of stuff[1]. I have a few raspberry pi's that serve as my "home lab" and that is all I really need right now. But I suspect the term took off because it gets a bunch of people like me to click.
[1] Not that I don't think it should exist, its just not high on my list of interests right now.
I wasn't before I read this thread!
I mean, my "home lab" are old computers from 2008 to 2016 and half of what I "do" involves simply testing what weird installations (eg netbsd) I can get up and running. It's for tinkering -playing; the "lab" part coming from experimentation: "What happens when I do this...?"
I agree when it comes to the definition being to narrow. Then again, I mostly engage with it on /r/homelab and the HN attitude seems to be much more restrictive...too restrictive for my tastes, personally.
The humor is a little goofy in the best way; the structure is detailed, but still conversational. It just feels very… personable.
Like — computers are so goddamn neat. We should all remember to have fun with this stuff sometimes.
To paraphrase the first item in the HN guidelines:
"anything that gratifies one's intellectual curiosity."[0]
You can get to explore configurations and scenarios you might never be able to do at work. But it's not about work, it's about scratching an itch.
> The most egregious factor of this hobby is that you are often buying equipment that costs enterprise prices
If you know where to look you can pick up enterprisey kit cheap. It may not be the latest and greatest, but it's probably good enough to play with.
Just because you don't understand the attraction of this pastime (or any pastime) you don't get to judge or complain about folks who enjoy tinkering.
Once upon a time I built out a home lab to run a piece of software called Zebra to learn about BGP4 and CIDR. It was about scratching an itch. If energy prices in the UK weren't so utterly bonkers at the moment then I'd love to build out a new lab to mess about with some stuff. Again, to scratch a curiosity itch.
I have a home ESXi server which I use to host all my own stuff. I don't consider that a lab as such. It's just my home production infrastructure.
And then I have another ESXi which hosts the stuff I'm testing for work, and only work.
In my work home lab I have full access which I don't do even in the testing environment at work due to fragmentation of responsibilities. Or if I do I step on people's toes when trying stuff out that belongs to their realm. Most things are interconnected so testing something at work usually gets slowed down by the need to involve other people who don't always have time or interest. That's where my own lab comes in.
It's also a lot faster due to not needing VPN and having to deal with servers halfway across the world. And not having to deal with the evil SSL-MITM Internet proxy at work saves me SO much time.. Of course that work needs to be done to put it in production but a lot of stuff never makes it to that stage and then figuring how to smooth things with the proxy is just wasted effort.
These days, the nomenclature survives, but the facilities are often virtualized or remote eg https://learningnetworkstore.cisco.com/cisco-learning-labs
I built mine in 2018 and I paid $65 each for two E5 2670 v3 ($1600 MSRP, 2014 CPU), and $300 for a dual socket motherboard with 128GB of ram. Yeah it's not the latest and greatest but it's been going strong since then.
I could say the exact same thing about however you decide to spend your weekends.
Thats quite a lot of people, personally i would rather be planting trees, it i could pay the bill with that
I don't really know if I qualify though. All that's on a 2011 Dell, a 2-bay Synology nas, and a UPS I found on Amazon. I have zero urge to go out and buy a rack; it works great as is.
And learning also fits into this: like the computer version of a school chemistry laboratory. Setting up computers to perform tests for certification.
For what it's worth I agree with you about the equipment -but I read on reddit a lot of /r/homelabbers getting equipment for work so that would make sense.
I don't work in the field so for me personally it's more a question of wanting to experiment with software while not throwing away perfectly working (but old) computers.
I'm not sure what the difference is between this and any other moderately expensive hobby.
As far as the name, I think it comes from the term "networking lab" that netops actually use to trial out network topologies before deploying them in the field. It's a home networking lab, that's all.
:)
My home network is an Eero. The network engineer’s kids have no homelab, as it were.
Occasionally, I get the urge to build out a proper homelab but quickly realize I don’t want to come home and deal with the same stuff and put the Eero back in.
It’s not that I don’t like learning and experimenting, it’s just that I get to do that all day at work and I need some work/life balance.
But, I still love seeing these posts and commenters chiming in with their own configurations.
I luckily had a mentor that let me bend the rules and bring stuff home. If you are a mentor you should also bend the rules for noobs.
In the end, my network at home works better, has higher availability and, for the times that it doesn't, I can quickly resolve the issue and get back to the enjoyment of my home versus being frustrated as to why some consumer-class thing is not working yet again.
I learned a ton and can now actually understand what's going on under the hood!
Since then though I definitely have simplified it a lot because it was a lot to manage but it was an excellent learning experience.
This is my long winded way of saying, network engineers don't get enough praise - that shit is actually really damn difficult to get right at scale!
In the end though, I got fed up with my consumer gear (Orbi mesh system) never being very reliable, so had the house wired for Cat 6 and shoved a UDM Pro, USW-Pro-48-Poe and 3 U6 Pro access points in and it's been far more stable and problem free than the Orbi, to the point where the kids have stopped telling me "Dad, the wifi sucks!" and needing to reboot the whole thing, which I had to do at least once a month previously.
I used to have a FreeBSD server running pf and jails to partition out services. Now I simplified everything with a little tiny Ubiquti cube that I use an app on my phone to control.
It is all powered by Ubiquiti networking gear.
VMware VUG is providing free and legal VMware licenses. I get to run VSAN, on top of it I deploy Kubernetes and I can decide if my pod wants to use poor performance VSAN disks (like EBS on AWS) or faster local NVMe‘s.
These are just couple of things I can make at home. Everything started as an experiment and it started to look like a replica of what we can have in a datacenter, but without a SLA.
First of all, all these stuff is extremely fun for me. Secondly, I won‘t go into details but as a cloud consultant, this thing I started as a hobby became one of the big boosters of my career. I never expected such an effect but I am extremely impressed by this.
- network switches: if you want enterprise features, low price, higher stability.. and you are willing to deal with painful configuration: check out Mikrotik - mentioned servers consume quite some electricity, check out this Dutch server thread on how to have efficient server for inspiration: https://gathering.tweakers.net/forum/list_messages/2096876 (use Google translate) note that the Framework main boards also look quite nice as server - also check out k8s at home template: https://github.com/k8s-at-home/template-cluster-k3s it uses GitOps to setup services which has proven very valuable to me by messing with configurations. I also build a search for helm chart releases: https://whazor.github.io/k8s-at-home-search/ - if going cluster mode: having a NAS for storage is more stable than using kubernetes storage solutions
All in all, I learned a lot about servers and networking by doing this as a hobby.
Attitudes toward dust ingress in particular delight me. Look inside any old Dolch data acquisition unit or tire alignment computer from the 1980s and find minimal reason to be horrified.
To think a home environment would necessitate anything special is excessive.
Pizza restaurants,OTOH.
[1] https://en.wikipedia.org/wiki/Electrical_discharge_machining
Don't get me wrong it's nice to do this stuff and I'm not knocking people for spending so much time on it but when you spend your day at work logging into systems and fixing broken junk the last thing I want to do is come home to a dead network or vm issue to solve.
https://www.reddit.com/r/HomeDataCenter/comments/ktz6yo/my_s...
I added 20KW of solar as well to offset to power usage.
It is interesting to see what other people have built, why they made the choices they did, and what they are using it for.
At first i Had a stack of 2-3 old laptops running stuff. Now I just use 4 raspberry pi 4s, with 8gb. It lives in a cubby in my office desk. Low energy, low maintenance and works really well. Got rid of the laptops.
My network is the TPLink Omada stuff, sittign on some shelves in our family room. Everythign is wifi and I run a 2nd ap on the opposite side of the house using MoCA 2.5 to connect them over the coax in the 2 rooms to improve performance.
I use https://rpilocator.com to monitor stock.
Basically nothing actually rack-mounts in 10.5” so I currently have things cable-tied to shelves, but I’m planning to 3d print some adaptors.
My setup: https://i.postimg.cc/xTpLPFQD/image.jpg
For mixing small gear with it, there's a lot to be said for rackmount shelves, including with zip ties or velcro.
For people that really only want something small, I've seen people 3D print out of plastic, build out of wood, IKEA desk organizers and furniture...
That's the "one big box" version of the 3/75 and pair of Sun shoeboxes that I once bought for home. (Bought used, from the local Sun office, which would occasionally email out of list old gear to sysadmins at customer sites, but for quite a lot more money than $25, unfortunately.)
Hopefully the 3/180 isn't as crazy-loud as the big old 4/390 that my employer was still using for email and some NFS.
Giant box to the right of the desk :) https://i.imgur.com/SpMMPXQ.jpg
(Currently have two Cyberpower rackmount units, which don't need as much rack depth in my living room, but I kinda expect something industrial-sounding like AMERICAN POWER CONVERSION to be more reliable than cheesy-sounding CYBERpower. :)
Newer stuff is sometimes a little better about ramping down at idle, but heck, some server fans will use more power than either of my desktop computers at idle.
But everyone please be careful not to do much keyboarding work without your keyboard and monitor straight in front (as well as observing other usual ergonomics advice).
That is not my experience. Most of the time it works just fine if researched well in advance.
The problem is just the impact for the few times it does not go well. That's why you have test environments. And test groups in production for the first rollout.
My home lab is more to try out the benefits of different options, especially as I work for a huge multinational. Admin rights are split among different teams and sometimes the others don't understand what I need. It's really helpful to have full access to my own lab so I can show them what I need from them and why it works best a certain way. Without access that's hard to figure out.
2U server with 6 4TB drives (ZRaid 2 leaving 16TB of usable space) running on an Asrock Rack WSI 246 mini itx board, a Core i3 9100T (35w, can be configured as a 25w part) and 32GB of ECC DDR 4.
It idles around 20w and going full pelt with all disks running at around 40-50w.
This is running UrBackup for the computers in the house to backup to, SeaFile for the phones, a Minecraft server for the kids, Plex, Wireguard (for remote access) and InfluxDB for collecting metrics.
Whilst playing with ex enterprise kit would be fun, the noise, and the power consumption, just isn't worth it to me.
* HP ProCurve 2910al-24G J9145A 24 port Gigabit switch (managed switch, €47)
* HP DL380G7 with 2xX5675 @3.07GHz, 128GB (ECC) RAM and 8x147GB SAS drives (€450)
* NetApp DS4243 (24x3.5” SAS array, currently populated with 24x650GB 15K SAS drives (4 as inactive spares), €400)
* the mentioned airflow divider
* 8 drying frames
It is managed through Proxmox on Debian and runs a host of services including a virtual router (OpenWRT), serving us here on the farm and the extended family spread over 2 countries. The server-mounted array is used as a boot drive and to host some container and VM images, the DS4243 array is configured as a JBOD running a mixture of LVM/mdadm managed arrays and stripe sets used as VM/container image and data storage. I chose mdadm over ZFS because of the greater flexibility it offers. The array in the DL380 is managed by the P410i array controller (i.e. hardware raid), I have 4 spare drives in storage to be used as replacements for failed drives.
The rack is about 1.65m high, it looks like this (here with the old D-Link switch (now deceased) and minus the DS4243 array which now sits just above the air flow divider):
In the not-too-distant future I’ll replace the 15K SAS drives with larger albeit slower (7.2K) SAS or SATA drives to get more space and (especially) less heat - those 15K drives run hot. After a warm summer I added an extra air intake + filter on the front side (not visible on the photos), facing the equipment. This is made possible by the fact that cooling air is pulled through the contraption from the underside instead of being blown in through the filter(s).
I chose this specific hardware - a fairly loaded DL380G7, the DS4243 - because these offered the best price/performance ratio when I got them (in 2018). Spare parts for these devices are cheap and easily available, I made sure to get a full complement of power supplies for both devices (2 for the DL380G7, 4 for the DS4243) although I’m only using half of these. I recently had to replace a power supply in the DL380 (€20) and two drives in the DS4243 (€20/piece), for the rest everything has been working fine for close to 4 years now.
On the question whether this much hardware is needed, well, that depends on what you want to do. If you just want to serve media files and have a shell host to log in to the answer is probably ‘no’, depending on the size of the library. Instead of using ‘enterprise class’ equipment you could try to build a system tailored to the home environment which prioritizes a reduction in power consumption and noise levels over redundancy and performance. You’ll probably end up spending about the same amount of money for hardware, a bit more in time and get a substantially lower performing system but you’d be rewarded by the lower noise levels and reduced power consumption. The latter can be offset by adding a few solar panels, the former by moving the rack to a less noise-sensitive location - the basement, the barn, etc.
As to having 19" rack equipment in the home I'd say this is feasible as long as you don't have to sit right next to the things. Even with the totally enclosed, forced-draft rack I made the thing does produce enough noise to make it hard to forget it is there.
Alas, it's not the same person.
That said, are there “Cloud” versions of this? Can I get my feet wet that way?
I’ve been concerned about this too. Initially I wasn’t worried, but in some arguments some people have warned me that even a single rack can incur significant monthly bill increases. I have a lot of retro hard too which I imagine is less power efficient.
You might use less power if you have separate low power devices as opposed to a single (large) server though, so that’s one consideration. Another those is what it is you hope to accomplish. If you want to test gear or hardware that you can’t virtualize, then a rack can be handy. But even things like pulling out hard drives from a running server can be simulated with a VM in some way.
But it’s easier to brag about having a rack than a really elaborate VM configuration.
Otherwise, as I missing something?
The only thing I feel I'm missing over over a real server would be an IPMI so I can boot it remotely in the event of a power loss. There are solutions to use a rPi for that which I plan to look into... some day.
Let's be honest, it's frighteningly easy for a rack to turn into a giant ball of wires too. See r/cablecore for examples.
It’s been years since I messed with “physical” networking stuff, but it wasn’t complicated. Complex, yes, but not complicated.
If you’re a “complete noob” and want to level up this skill, then learning what all the equipment listed does, why you’d want it, and how to configure it is an excellent way to go about it.
* Don't underestimate the number of services you can having running in containers on a Raspberry Pi 4B with 8GB of RAM, sipping just 2.1W (yes, that little!) at idle over PoE or as low as 1.35W over WiFi powered by USB-C. With distributions like DietPi you can have a mere 10 processes at startup (using just 44MiB of RAM), but still all the flexibility in the world.
* Also don't forget you can cluster them. Want Kubernetes experience? Perfect opportunity to have a cluster than pulls at most like 17W under extreme loads.
* Have an old laptop? Fantastic. Great second life for it. You get the bonus of an integrated keyboard, UPS and display if and when you need it, even if it's normally headless.
* Don't forget Mac Minis and 1L form factors. I have an 1L form factor (that weighs about 1.3kg) 8-core Zen 3 Ryzen 7 PRO APU that boosts to 4.6GHz, with 64GB of memory, a 1TB SSD and 2.5GbE that sips a mere 10W at idle, and pulls maybe 45W at full bore. Its silent. takes up as much space as like 25 paper napkins, and I can pass the GPU cores through to VMs.
* Don't forget hypervisors as well. For truly ephemeral stuff you can just spin up virtual machines on your main desktop or laptop -- just stuff more RAM in it.
-- ----- -----
What's great is you can put the power sipping stuff basically ANYWHERE. It can be easily accessible, but entirely hidden from view. It makes no noise. It's not spewing heat or demanding cooling. There's no blinking lights. You're not driving up the electricity bill. Yet you can do very real work on them.
For cost? Raspberry Pis aside (sorry, use rpilocator!) check eBay or other marketplace completed listings to establish a price floor, and just be patient and only grab quality stuff that approaches the price floor. Nothing better than getting new (but open box) or barely used modern hardware for peanuts (60-70% off) because the seller is impatient or ignorant.
Had x300 for quite a few years but have started playing games again, so decided to trade in for a desktop...but will probably go back if RDNA 2 is good/x300 gets updated for it (imo, the iGPU on the 4750g is terrible...RDNA 2 is way better). Great for running VMs on though, can take a Noctua NH-L9 so is quiet, far less space-intensive.
I've been disappointed in the general reliability, terrible IO, and heat/power of the RPI4. I'm getting rid of them in favor of second hand thin clients.
> old laptop
> Also don't forget you can cluster them. Want Kubernetes experience?
Just don't be thinking about mixing ARM and x86 in the same Kubernetes cluster. Like even if you technically can run a multi-arch kubernetes cluster, it's a major footgun. Docker's multi-architecture support leaves a lot to be desired and is a real pain to work with.
What computer is this, if you don't mind sharing? Sounds great for getting 1 or 2 and hosting a lot of VMs.
The ProDesk 405 G8 can take a 35W RYZEN 7 PRO 5750GE. This includes AMD DASH (IPMI), and the Ryzen Pro has a few extra security features from EPYC/Threadripper. I got mine shipped for just $665 including tax, and did the SSD (1TB 980 Pro) and memory (64GB DDR4-3200 CL16) myself for another $300ish.
If you get three of them, you can have a pretty fantastic Spark cluster that consumes only about 100-120W! But yes, they do XCP-NG or Proxmox clusters extremely well.
The EliteDesk 805 G8 can have the 65W Ryzen 7 PRO 5750G, has a copper heatsink (instead of aluminum), has a 2nd NVMe slot, and can be shipped with a 2.5GbE port (you can order the part for the ProDesk 405 G8). Though if you get the 35W 5750GE, you can have an NVIDIA GeForce 1660Ti 6GB installed as well.
Since both use Ryzen PRO processors, they actually support ECC memory as well. I'd argue they may be the best overall 1L form factor systems on the market, especially in terms of performance per watt. The only other thing I'd ever want is 10GbE instead of 2.5GbE -- though in the EliteDesk 805, you COULD get an NVMe to 10GbE Ethernet and install it, just need to make a faceplate to properly mount the port on the back of the chassis.
Originally I opted for the 5700GE (apparently slightly faster than the 5750GE) thinking I wanted the most power possible in the box. But my new thinking is I really want a 5300GE to pull out and put in an AM4-based silent HTPC for the living room, then upgrade the HP with a separately purchased 5700GE. QuietPC seems the only vendor selling these "GE" 35W models -- but not any 53X0GE parts. In other words buying one of these non-vendor-locked HP minis and ripping them out might be the only way on Earth right now to get one's grubby hands on any 53X0GE.
Also, are you sure there are two M.2 slots on the EliteDesk G8's? I know they are on the G6 but haven't found confirmation the G8's continue the tradition.
It's generally a really good form factor for a small server. The only things most people sometimes wish they had in them is 2.5-10GBE instead of the stock 1GB, and formal ECC support.
It runs esxi that auto boots my "workstation" vm on startup. This vm has my GPU, nvme, and usb passthru.
This way I can boot my workstation "on metal" if needed, but the vm works great. I then manage my lab infra running on the same machine via vCenter html5 ui.
https://i0.wp.com/williamlam.com/wp-content/uploads/2022/04/...
https://pbs.twimg.com/media/E5sm0cjXoAIfLM9?format=png&name=...
Here are some other reasons:
* Best support for PCIe passthrough, easily toggle PCIe devices from UI without reboot.
* Best networking with virtual distrubuted switches - more intuitive to manage and use
* vSAN integration with k8s for block volumes is great, including volume snapshots. Also supports RWX (nfs) volumes
* vSphere api is a first class citizen with cloud native tooling such as terraform, ansible and packer. It also has a great golang-based cli via govc.
With all that said I actually still experiment with kvm (kubevirt, harvester etc) by running it nested inside esxi which works exactly how you would expect. Much prefer vSphere as the base of my lab though.
ESXi remains one of the most ridiculously and stupidly stable platforms of all time. Someday someone will find an ESXi host with a twenty year uptime. Not because you should, but because it can.
-- -----
A 12-core Ryzen 9 5900X is now $395 everywhere in the US, though a patient soul can get one on new eBay for $370-375 before sales tax (but including shipping), best $350 used.
DDR4-3200 CL16 from top brands isr $3.50-$3.65/GB new, or under $250 for 64GB. Again, a patient soul can save more still and get it closer to $200. Want 32GB? Cut those prices in half.
So ~$530-650 new with sales tax and shipping, with a floor of maybe $455-575. For a tremendously capable processor and 32-64GB of memory.
There's very few consumer options at 2.5 / 5 GbE; you can usually pick up cheaper 10 GbE gear (but on older standards, non-multi-gig compatible).
Indeed. I'm ordering my 8 Gbit/s "fiber to the home" router and it should come soon (45 EUR / month). The router comes with one SFP port doing 10 Gbit/s (!) while the four other ports are regular 1 Gbit/s ones.
I use it to connect my NAS, my vm server, my Mac, and uplink to my nortel. Works nice.
Most PCs where I work run through 100 Mbps switches.
Your experience is very far from the average.
They also have a 2-Gig ($120/mo) and 5-Gig ($300/mo); I'm waiting for those to drop in price and I'll definitely jump on-board (most of my home network is on 2.5GbE with a few 1Gb segments).