Hacker News Reader

Comment by chillfox | Hacker News Reader

Setting up NFSv4, I spent two evenings on that before abandoning it.

Virtual network for a bunch of jails with nat. I wanted exposed ports on the host machine mapped to a reverse proxy jail that did tls termination before forwarding to other jails on a virtual network (while allowing the jails on the virtual net outgoing internet access).

Several applications that were apparently built for Linux, so they did not compile on Freebsd (and no alternatives available).

Getting newer versions of software. Often the version available in pkg is several major versions out of date.

Documentation for how to do things in general. Yeah when there are docs/tutorials they can often be of high quality, but frequently there is just nothing or the available docs does not cover the edge case you are trying to solve.

View on HN

> Virtual network for a bunch of jails with nat. I wanted exposed ports on the host machine mapped to a reverse proxy jail that did tls termination before forwarding to other jails on a virtual network (while allowing the jails on the virtual net outgoing internet access).

Jails are old, and there's a wealth of outdated documentation around which makes it confusing when trying to understand how things works.

In case someone else is reading this, it's not different from what Docker does automatically for you.

I typically use vnet (which gives the jail an entire virtualized network stack, something like macvlan?) and either use NAT or bridges for connectivity. NAT for machines with only one public IP (like VPSes) and bridges for homelab setup where I want a more visible setup.

When setting up NAT I just use PF as my firewall, and assign static addresses to each jail.

Virtualized network stacks in Linux are network namespaces with a difference of not being tied to a specific "prison". So every Docker network regardless of type creates one.

I never had any doubt that it was possible, I just couldn't figure out how to get it working and in the end decided to not bother with jails.

Ah, understandable. I remember being slightly confused when starting to use jails again, after being used to the near magical setup of Docker/etc on Linux. Like manually setting up bridges and VLANs to make it work like I wanted? WTH.

Now I dislike using Docker for similar tasks, as I find it to be more work than just doing the same setup with VNET, bridges and VLANs. I'm a fossil and I want to run different services on different VLANs, which in Docker lingo is considered "legacy application setup" (looking at the macvlan documentation).

>Setting up NFSv4, I spent two evenings on that before abandoning it.

That's really a bit of a pain (lack of a good tutorial/doc)

>Virtual network for a bunch of jails with nat.

I solved that this way: Create'd a vnet haproxy-jail (the tls-terminator and "forwarder") all other jail's a "non-vnet" with a internal IP.

But you can do it with just nat'd jails and haproxy as the terminator...no problem, do you need some pf examples?

>Getting newer versions of software. Often the version available in pkg is several major versions out of date.

Same on Linux-Distros, freebsd is pretty much on par:

https://repology.org/repositories/statistics

>Documentation for how to do things in general. Yeah when there are docs/tutorials they can often be of high quality, but frequently there is just nothing or the available docs does not cover the edge case you are trying to solve.

Can you give some examples so one could do something against it? NFSv4 with kerb and ldap is one..i know ;)

With jails, I could really use a tutorial that goes through a standard proxy/app/db setup (nginx, wordpress and mariadb maybe?) that does require multiple ip addresses on the physical network. So request comes in to the host on port 80/443 -> proxy -> app -> db. All jails can do outgoing connections for things like sending email, downloading updates, etc. The db and app are on a network with the proxy and only the proxy can be reached from the outside. So what would be a pretty normal setup with Docker.

And yes, firewall config should be part of that rather large tutorial, but it I really feel like a complete tutorial is missing. The ones I have seen only cover a part of it and it's left as an exercise for the reader to figure out how to glue them together.

Some of the other issues I had with incompatible dependencies would be solved if I know how to use jails correctly.

How to get a Linux jail up and running (Alpine Linux).

ZFS + NFSv4 + Kerb + ldap would be great. I ended up solving that particular problem by using minio instead, because it was so much easier to setup, but I would like to know how to get nfs working for future things.

> Several applications that were apparently built for Linux, so they did not compile on Freebsd (and no alternatives available).

There's a PR hopefully landing soon in nixpkgs to add freebsd building: https://github.com/NixOS/nixpkgs/pull/82131

I was wondering if there's a builder at some point, it may improve the compatibility for the whole ecosystem. There's definitely a lot of fixes going upstream from darwin build failures.

Nix is cool tech, but really suffers from a lack of good docs. I tried to use it for dev environments, but it's a struggle and asdf just has the user experience nailed, so I abandoned nix and went with asdf instead.

> Getting newer versions of software. Often the version available in pkg is several major versions out of date.

The default settings for pkg use the quarterly branch. Remove the comment from the line for using the latest branch in /usr/local/etc/pkg/repos/FreeBSD.conf.

Thanks

Reply on news.ycombinator.com