Hacker News Reader

Comment by sgc | Hacker News Reader

I have been looking into setting up my first Proxmox box, here is my take as a newcomer.

I wanted to do what I think is a very basic and very common setup: Modem > proxmox box > OPNsense VM > physical wifi router via onboard 10Gb NIC + internal network VMs like OMV etc. The goal is to add a full network filter via OPNsense, and allow access to a media sever and backup etc from the internal network.

I see no OPNsense, OMV script is basically contra-indicated because it should be a VM instead of the LXC container, and I don't see any glue scripts to get VMs talking to each other, which is an important part of Proxmox configuration. So it looks like there is room here to get some basic setup scripts for a simple home server either improved or added to the collection.

View on HN

No it isn't basic and common (it is for me but perhaps not for you and certainly not for most people)

OK, so you want to virtualise a router and firewall. That's fine. I have deployed roughly 200 pfSense firewall/routers as VMs and physical boxes and OPNSense is similar, so I can probably help.

At a minimum you will need two physical interfaces (one will actually do but you will need to know what you are doing!). You need "WAN" and "LAN". OPNSense is still FreeBSD based, I think, so it will not run in a L[inux]XC container for obvious reasons.

Your last paragraph seems rather confused. I don't know what you mean by "glue scripts". VMs communicate via networks

I suggest you try a few experiments to get to grips with virtualisation properly and then move on from there. If you swing by the Proxmox forums with specific issues we'll try to help out but in the end you need to dive in full on ... or not.

I run proxmox and have set up VLANs.

The router port to the proxmox machine is set up for tagged packets that isolate incoming/outgoing traffic.

After that my VMs and Containers are easily set up to "live" on one or more networks.

For me the firewall rules on the router determine what traffic can be relayed between vlans through the router.

I'm pretty sure you could set up opnsense running in a container or vm to do the same thing, selectively passing traffic from one vlan to another.

i have a similar setup with a PM box and a Ubiquiti Dream Machine Pro. i provision VMs with a Terraform provider, have a script that processes Terraform outputs into an Ansible inventory INI file to handle configuration. i find it pretty straightforward and could take it further by scripting my VLAN setup but changes so infrequently i don't mind doing it manually.

There is no OPNSense script I think historically in part because any misconfig could expose the Promox instance to the world. It is easy enough for advanced users to spin up a VM with the ISO. There has been a request for a OPNSense script made recently.

I agree with OMV. It certainly can be used as is, but not usually how people want to use it. A note was added to the script a few days ago.

> I don't see any glue scripts to get VMs talking to each other

There is a Tailscale script which technically helps them talk to each other (over Tailscale) :)

The scripts are designed to setup self contained LCX containers. We are trying to avoid building our own k8s.

Great, now I am down the tailscale rabbit hole and just have to use it!

I think I will stick to using proxmox virtual ports to create my network so I can more easily only stick to individual device registration in tailscale and save on that overhead when I'm home, but then also add tailscale /headscale into the mix somewhere so I can tap in via VPN when I am out of the house.

Tailscale and OPNsense are more difficult to get working together due to conflicting project goals (one blocks well, the other opens up well), but it looks like it's worth it to me.

I use Proxmox with an OPNSense VM and have multiple NICs - one is dedicated to the fibre ONT. I also use an external wifi mesh. I have a couple of other vms (unRaid hosting Dockers with sata card passthrough for legacy reasons and a vm for Home Assistant OS) and lots of other LXCs. It works superbly.

> I don't see any glue scripts to get VMs talking to each other

I'm confused by what you mean here? Don't they just use the network like any other computer?

I haven't had to do any special configuration to get my VMs to talk to each other.

I am not perfectly informed, but in my case, OPNsense would need to be the only vm with access to the incoming NIC port, and all other VMs and the router would need to use virtual network interfaces only coming from OPNsense for incoming. The router would be the only device with direct access to the outgoing NIC port. None of that seemed incredibly difficult looking into it, but still, it was the type of recipe I was expecting when I saw "Proxmox scripts".

And of course this means that the Proxmox box as a whole should have similar hardening to a typical web server, with minor tweaks to allow residential traffic on various other standard ports. So that hardening would probably be another script I would like to see (I don't know what all the proxmox scripts in the first section do).

VMs already use virtual network interfaces, which are by default bridged to `vmbr0`, a bridge that proxmox creates by default which is also bridged to the hardware NIC. For your use case, you simply want to create a second bridge, e.g. `vmbr1`, which is not bridged to the hardware NIC. You would then assign two virtual NICs to opnsense, one on each bridge (WAN and LAN, essentially) and then choose `vmbr1` as the bridge each time you create an "internal" service behind opnsense.

Since selecting the bridge for a service's NIC is part of setting up each service, the only thing such a "glue script" would be doing is creating the `vmbr1` bridge. That's already a one-liner.

I was looking at a proxmox/(pfsense/opnsense) tutorial the other day. They recommend binding the WAN interface to vmbr1 (or anything other than vmbr0) since VMs are created with their ethernet bridged to vmbr0 by default. This configuration is what most people want so it'll be a little less work setting up networking.

VMs usually have their virtual NICs connected to a bridge interface on the host (like a virtual switch) so they can communicate. Proxmox creates one up by default that is also bridged to the physical NIC you set up for management when you install it, so it just works.

In the router case, you'd likely want this default one to be the 'internal' network and have a separate interface (either physical or VLAN) for the WAN.

Reply on news.ycombinator.com