Homelab: Intel NUC with the ESXi Hypervisor
henvic.dev
henvic.dev
* dual NVME drives for all NVME vSAN
* dual nics work out of the box for ESXi
* dual thunderbolt 3 ports to enable 10GbE via TB3 adaptor or PCIe expansion
* Linux friendly AMD GPU that has no problems being passed through to a nix or win vm
* low profile and easily stackable for a rack shelf
* Supports 2x32GB SODIMMs
* Power efficient
* Low noise
Love these things!
It's got i9/Xeon options, 3 NVME drives, and even PCIe x4 and PCIe x16 slots. It's bigger but still quite small. The PCIe slots are the killer feature to me.
[1]: https://www.intel.com/content/www/us/en/products/boards-kits...
* XCP-ng (based on xenserver)
* FreeBSD bhyve.
(so that you boot up your machine into one of the virtualized OSes)
Also have seen some folks who have used a Pi or a Chromebit to remote into any machine they choose.
You can mix and match normal VMs with LXD containers, and it does quite well for single physical hosts or small clusters. Past some scale, I'd go with something else, but it's great at the < 20 physical host range. It's a fairly unsung open source area. OpenVZ used to be a semi for-profit, open source/ but we want money, competitor here. Guessing they are mostly dead now. (Not celebrating that...they did a lot of good).
Yes. Proxmox is built on Debian (or was, last I played with it) and you can actually install it on a Debian host, if desired.
> (so that you boot up your machine into one of the virtualized OSes)
No.
You can get good, power efficient pfsense hardware for around $200 USD. A few bonus notes:
—You could spend a little bit more and get hardware that's good enough to add a virtualisation layer to your dedicated router.
—If your uplink is less than 300 megabit, and you have a managed switch with VLANs, your pfsense box doesn't need to be dual NIC. This could open the door for cheaper or re-purposed hardware.
Upgraded to the latest pfSense version a couple of days ago and the total downtime was in the order of minutes.
It's a bit too early to say whether it's a good idea or not; the main concern I have is that you have two points of failure instead of only one. Other than that worry it has been very smooth.
That, and CARP on VMware is ... tiresome.
That said, there's value in having your router/firewall separate from other systems (virtual or otherwise), which will substantially reduce complexity and make it less likely that you accidentally make a system accessible on the wrong side.
In particular, qemu and apparmor don't always play nicely together and the error messages you get by default don't tell you that it's an apparmor problem.
Importing a VMDK package was also tricky, I ended up having to get familiar with some XML incantations to get the machine to import properly. Feedback from qemu or its friends was totally obtuse and unhelpful there too.
I got home network and wanted KVM to control which VM gets which IP, but I couldn't make them accessible from outside - without static IPs on VM side. When I use virtual bridge, KVM cannot set the IPs, but at least they are accessible from my home network.
It doesn't have the same hype as the others but far more open. Biggest drawback is that the configuration software is windows-only (you can get it running in wine but it is a bit hacky, also xen-orchestra is a free (for personal use, if you build from source) web-configuration utility that one can use).
Would not want to be at mercy of vmware. Also don't want to invest time into a system with such restrictions on use.
Proxmox might be a decent alternative but I have a few systems and the cost would be prohibitive for hobby use.
Not sure if a gimped shareware-like(?) option sounds that appealing.
Proxmox is great as long as you have the mental fortitude to ignore the tickler that pops up on login if you're on community support only.
If web works with base ESXi great! If not then I'm not interested.
I just don't work on a lot of companies in the position of needing VMWare anymore. Either they're all in on AWS or they're too small to justify the expense.
I haven't actually used Proxmox but had X11/Xquartz on macOS, following enough to manage them. You don't even need to open X manually. It gets started when there is a connection.
ssh -X user@host -- virt-manager &
Also getting started on KVM on Debian is as easy as: apt install -y libvirt-bin virt-managerYou have to fiddle with template images but it's kind of a tedious task.
There is also PhotoStructure [2] that I like the look of, but PhotoPrism is OSS and already integrates machine learning for identifying objects in photos... yes I know I can't expect great performance out of my N54L on that front!
But there is actually code, tucked away in the hard to find github link: https://github.com/photoprism/photoprism
Unfortunately, it requires docker, and cannot be installed standalone. That's a deal-breaker for me.
I discovered it a few months ago when I was searching the web to see what is out there. Previously to moving to iPhoto (regretted a lot), I used digiKam and I enjoyed it.
* I don't want to risk reading something I like there and ending up copying the idea or writing similar code unwillingly, as this license is viral.
Thanks for showing me PhotoStructure!
I used iPhoto for a while and am actively considering buying a used one to run it on again for family photos. What didn't you like about it, and is there anything that can take the metadata from iPhoto and translate it?
So, don't? Just pretend that it's a proprietary application with no source that just drops from the sky as a binary or docker image. Can you not get what you want as just a "normal" user?
If you've signed up for beta access, know that I'm sending out invites in batches, and hope to send out another this week, with the release of v0.8.0 : https://photostructure.com/about/release-notes/
(sorry if you've been waiting a while!)
Briefly, there are 3 builds of PhotoStructure: a desktop build (packaged with Electron), a Docker image, and a git repo you can pull, `yarn install`, and run directly. More details about why I felt the need to quit my office job and work on it full time are here: https://photostructure.com/about/introducing-photostructure/
If you've got any questions, feel free to ask them here or email hello@photostructure.com.
Storage is an NFS mount from my Synology NAS.
The only reason I would really care to have a hypervisor is because then I can do more from Terraform.
I use Terraform to configure my DNS, my Ubiquiti gear, and the Swarm cluster but that leaves a gap where I need to do something to manage the actual CentOS machine. There isn't much to manage (users, SSHd config, SSH keys, packages such as vim, docker, and htop, and then NFS mounts) but the less I have to manage myself the better. Just don't think it's worth adding a whole hypervisor just to pull possibly some of the networking into Terraform.
Anyways, NUCs + Docker Swarm = great win.
* Client: 192.168.0.8/29 (whatever the smallest subnet is I can use that includes my NUC)
* Privilege: Read/Write
* Squash: No mapping
* Asynchronous: Yes
* Non-privileged port: Denied
* Cross-mount: Denied
This is fairly out of date (I keep my own stuff in a private monorepo, this was a snapshot I posted for a friend), but here is the basics of a script to setup my NUC: https://github.com/regner/homelab-stacks/blob/master/scripts...
Of note, the "Adding NFS share to fstab" part doesn't actually do that...
I recently picked up some old budget Gigabyte BRIXs that will take on some other fun roles. (A 6th Gen Celeron and a 4th Gen i5.)
I have wanted to run an AD domain for managing my gaming PCs, but I had some difficulty with ESXi IIRC (probably also NIC issues, like in the article), and Intel has decided to be a complete tool about network drivers for "consumer" chipsets being installable on Windows Server. One of the perks of the old 4th Gen BRIX unit: A Realtek LAN NIC.
One of the things I'm sad about on newer NUCs is the removal of the LED ring. 7th gen NUCs in particular have a neat multicolor LED ring around the front panel that can be programmatically controlled... pretty neat for server status like uses. But I suspect they found it rarely used and so left it out for cost or something.
The network device is annoying - I couldn't get it to work on debian stable, but it is supported in Debian testing thanks to kernel 5.5
I also had problems with the HDMI video - It didn't work with the hdmi->dvi cable I got with either of my spare monitors, nor did it work with my TV and the existing hdmi cable I had there. I did get it to work by borrowing the hdmi cable from a friends htc vive.
Other alternatives include running pihole on one machine (or all machines for backup), or giving them static IPs in your DHCP server and then copying a hosts file to them all.
Then you pass traffic on certain ports to certain IPs if you want to be able to access anything from outside.
Or bridge those VMs on your macbook so it's a flat network, then use Avahi for mDNS so they all have .locals.
edit clarify it was $250/each
The CPU is roughly equivalent to a 6th or 7th gen i5, but with 8 threads instead of 4.
Installed proxmox with 1 VM and several containers. Couldn't be happier with everything. My wife can't even hear it under decent load in the living room.
Thing is, running all 16+ blades pushes my electric bill over $500/month, and sounds like a fighter jet getting ready to take off :-P
Having silent gear is totally underrated!
> I discovered the Ethernet NIC wasn’t working.
> More Ethernet issues
Likewise with 5.5, where enabling the ethernet is often nontrivial (e.g. injecting the drivers for the Realtek NICs).
VMWares website is utterly impenetrable and it’s like a form of hell trying to get out of the loops you end up in with licence agreements and download options.
Oh man, yes. The only way to download installation binary of something specific was a direct link to VMWare servers found somewhere on the internet.
Also, the Windows client was dropped a while ago and the Flash version is now deprecated.
At the time I bought it (2013?) it was far better value for money than any of the Intel NUCs.
I've always wanted someone to produce a NUC in a similar form factor, with good design but there doesn't seem much demand
And alas the current Mac Mini's are completely un-upgradable
Only negative I ran into is - I bought the bigger one with NVME and regular SSD. I couldnt fit a SSD in with NVME with heatsink :-/ Besides that this thing is snappy and uptime been perfect.
Currently running the following containers-
proxy [1] - Nginx proxy host for all ingress traffic, has a nice web interface and works with Let's Encrypt
pihole [2] - Primary DNS server running Pi-hole to block ads
cf-ddns [3] - Client to update Cloudflare records with my IP address (wildcard record *.myhomelab.dev which I run the proxy
hosts through)
unifi-controller [4] - UniFi Controller appliance
watchtower [5] - Keeps the docker images up-to-date
portainer [6] - Web interface for managing docker containers
guacamole [7] - Apache Guacamole remote desktop gateway
postfix-relay [8] - Open SMTP relay on my internal network which forwards everything to Amazon SES, makes email notifications easier
It's a great piece of kit and love the small form factor, it sits in my network comms cabinet and I've had no issues. I pass through a monitoring cable from a UPS in the cabinet to the CentOS VM which monitors if there's a power cut and shuts the ESXi host down safely.I initially ran PhotonOS with Docker but had some networking issues so switched to CentOS 7. I have larger, more powerful SuperMicro hypervisor hosts which run the bigger application containers.
[1] https://hub.docker.com/r/jc21/nginx-proxy-manager [2] https://hub.docker.com/r/pihole/pihole [3] https://hub.docker.com/r/joshava/cloudflare-ddns [4] https://hub.docker.com/r/linuxserver/unifi-controller [5] https://hub.docker.com/r/containrrr/watchtower [6] https://hub.docker.com/r/portainer/portainer [7] https://hub.docker.com/r/oznu/guacamole [8] https://hub.docker.com/r/simenduev/postfix-relay
I'm not currently doing it with my NUC servers, but it's a pretty good choice to do so, and ESXi is free if you don't need to, you know, manage it in any competent way.
* I'm going to fix this later today + some misspells.
- Any linux-based OS, Ubuntu in this case.
- Ansible / Terraform for task automation.
- K8s / Docker for containerization.
Given I have only 1 desktop + laptop, this option is good enough for me to learn cluster setup / networking / automation.