Hacker News Reader

Comment by nyx | Hacker News Reader

Ahh, /etc/machine-id, we meet again... Last time I met you, you were causing my cloned VM to get the same DHCP address as its original and blow up my VM network, despite libvirt's virt-clone utility properly randomizing vNIC MAC addresses, because netplan defaults to using you as the DHCP client identifier. If only we could meet under more pleasant circumstances.

View on HN

NixOS has a related problem in that you can't set the machine's MAC address without a hack:

      # Hack: Change the default MAC address after network but before dhcpcd runs
      systemd.services.setmacaddr = {
        script = ''
          /run/current-system/sw/bin/ip link set dev eth0 address ${macaddr}
          /run/current-system/sw/bin/systemctl stop dhcpcd.service
          /run/current-system/sw/bin/ip addr flush eth0
          /run/current-system/sw/bin/systemctl start dhcpcd.service
        '';
        wantedBy = [ "basic.target" ];
        after = [ "dhcpcd.service" ];
      };

If you don't do this, it uses some black magic to decide the MAC address based on various hardware, making system migration and maintenance a nightmare.

Is it NixOS doing it or is it e.g. systemd-networkd? (Sounds like something systemd would do.)

While I’m confused as to why the boot would need to decide on a MAC address (VM or cheap SBC without a burned-in address? the first case might be easier to correct from the outside), the general state of NixOS is that some things are very flexible while others only cover some common cases (ones that the original author needed to solve). Unlike a traditional distro where the package manager will complain if you replace distro-provided stuff, in NixOS it’s entirely possible to override parts that don’t work for you rather than paper over them with programmatic overrides like these. It’s not even hard to upstream your changes if you make them backwards-compatible, although the benefit can be limited because the testing is not particularly thorough so other changes may still inadvertently break them.

I gave up trying to find a root cause for this after a couple of days of rabbit holes and yak shaving. NixOS is simply too impenetrable once you fall off the happy path (which is unnervingly often).

The next time I rebuild this server, I'll go back to Ubuntu or Debian and use build scripts for "good enough" determinism. For the time being, I just run everything important in LXC and Docker containers on top of this delicately balanced NixOS hypervisor for as long as it'll last.

It’s the fate of every config generator, yes, although I find NixOS better than average in that respect (between Arch and Debian in how easy it is to figure out what the hell it’s doing to the underlying software).

Troubleshooting guides for NixOS are non-existent, but the system itself is not all that difficult to inspect: two things you can do is `nixos-rebuild build` without switching and meditate on ./result/; and inspect `(builtins.getFlake(toString ./.).nixosConfigurations` in `nix repl` (use import etc. if not using flakes) as that will include every derived setting down to the text of generated config files, not only those you specified explicitly.

But you might’ve just nerd-sniped me, we’ll see.

ETA: Looks like it’s indeed systemd-networkd’s doing[1]. For a static interface, setting `networking.interfaces.${NAME}.macAddress`[2] to the desired value should work.

[1] https://freedesktop.org/software/systemd/man/systemd.netdev...., see the description for MACAddress.

[2] https://search.nixos.org/options?show=networking.interfaces....

> Is it NixOS doing it or is it e.g. systemd-networkd? (Sounds like something systemd would do.)

Given that OP uses dhcpcd, it's unlikely to be systemd-networkd.

Good point. But something has to be setting those MAC addresses...

And it looks like, curiously, explicitly configured interfaces have their setup expressed as .link units even if networkd is not in use[1]. A comment[2] states: “.link units are honored by udev, no matter if systemd-networkd is enabled or not”.

It seems that .link units are nowadays interpreted not by networkd (which NixOS gates with useNetworkd) but by udevd (which it does not). The documentation for them (but not for udevd) even points that out[3] if you’re the kind of person who reads introductions: “link.link: A plain ini-style text file that encodes configuration for matching network devices, used by systemd-udevd(8) and in particular its net_setup_link builtin”.

[1] https://github.com/NixOS/nixpkgs/blob/65e07f20cf04f5db9921dc...

[2] https://github.com/NixOS/nixpkgs/blob/65e07f20cf04f5db9921dc...

[3] https://www.freedesktop.org/software/systemd/man/systemd.lin...

We (as far as I can tell) use some parts of systemd-networkd by default (as in, even if you haven't enabled it), as MAC addresses via `networking.interfaces.<name>.macAddress` are set through it: https://github.com/NixOS/nixpkgs/blob/65e07f20cf04f5db9921dc...

kstenerud: I assume you tried that option?

Edit: ah, you caught that in an edit :-)

virt-clone only takes care of the libvirt config but won't touch unique identifiers in your disk image.

/etc/machine-id isn't the only thing to worry about when duplicating a VM - think SSH host key, DHCP leases, various filesystem UUIDs, log files, MAC addresses in ifcfg-* files and udev-persistent-net rules...

You can use virt-sysprep[1] to clean up a disk image.

[1]: https://www.libguestfs.org/virt-sysprep.1.html

Cloning VMs is kind of an anti-pattern anyway, it’s absolutely full of this type of foot gun. Just use ansible or similar and build a new one.

or cloud-init

boot the clean base image and configure it on boot with packages, ssh keys, custom commands, etc.

Ansible has a non-trivial transition cost (speaking from experience here). Cloning VMs is a legitimate stopgap measure, because that’s sometimes the best you can do if available engineer hours are tight.

Ansible (as a proxy for infrastructure as code in general) being hard to implement is a warning that your setup is too convoluted. Cloning VMs is the high-interest unsecured loan of tech debt, and when that bill comes due it’s going to be much worse than spending a few days on some scripting.

Completely agree on all points, except for "few days". It took much longer than that, but partly due to learning Ansible as I went. The pain was still worth it, though.

Or better, cloud-init. It's same thing that cloud images to use to specialise VM "clones" to their environment, so is more likely to do all the right things. If it has a mechanism to detect that it's been cloned (eg. instance-id on EC2, etc) then it will deal with ssh key regeneration and everything else.

As another comment pointed out elsewhere you are not supposed to touch /etc/machine-id to control apt. It has its own Apt::Machine-ID you can change without breaking anything else.

That's almost worse in my opinion. If you're going to have unique machine ids, is it too much to ask that there's only one to deal with?

In the default configuration apt uses /etc/machine-id, it just has its own override (which is reasonable, imo).

Oh, perfect; yes, that's the best way to do it. (Best of both worlds, perhaps I should say.)

When I read what netplan is supposed to be I thought "FINALLY", when I saw who is making it (Canonical) I said "fuck, they will fuck it up"

Back in the NT4/Windows 2k days I recall having to do some extra steps to modify an identifier in the registry (or similar) when cloning Windows images.

Otherwise the clone would not properly register on the network. Perhaps it was only when speaking to the Domain Controller though.

IIRC the later versions of Norton Ghost, which was what we used, did this process for us automatically.

It's the SID you were changing, and it turns out it was never actually required: https://techcommunity.microsoft.com/t5/windows-blog-archive/...

Russinovich eventually pulled the tool from circulation.

Ah yes, that was it. I do recall we got error messages preventing the machine from working, and the guys spent some time researching before a solution was found. The error messages went away after changing the SID.

There are some conspiracy theories, fuelled by the fact that Russinovich only pulled the tool after being acquihired by Microsoft.

It's undeniable, in my experience, that the tool did help, despite all the swearing to the contrary. Making a leap from there to believing that it probably made it too easy to clone Windows machines in a way that Microsoft had no control on, and hence asked him to pull it, doesn't seem so crazy though.

You just needed to actually read the documentation and use sysprep (ideally with an unattend file). Just remember to image it before sysprepping since you can only run it a few times. There are a ton of things sysprep does that are really helpful and not handled at all by tools like that.

Microsoft doesn’t care if you clone systems, (why would they?) they care about the volume of help desk tickets created by doing it wrong.

> You just needed to actually read the documentation and use sysprep [...] There are a ton of things sysprep does that are really helpful

And a lot that are not helpful, iirc. It was probably 10 years ago when I had to deal with this, but my recollection is that Sysprep was messing with a lot of stuff - more than I needed.

> Microsoft doesn’t care if you clone systems, (why would they?)

Lol, licenses, of course. Nowadays they got a bit softer on the issue, but back then they were still very very twitchy about duplicating and virtualizing systems.

We have very different memories, then. The official docs were really good as far back as 2006 or so, they just required you to actually read them. Sysprep was certainly a powerful tool, but if you told it not to generalize and no OoBE, it mostly just reset the SID. Most people I ran into who had issues were using some random poorly written blog posts that were really just content farming for ad money.

Huh? I’ll grant you that Microsoft was picky about being paid for their software, but they produced a ton of tools to support cloning and duplicating. License compliance was handled with audits and CALs, not some weird cabal of anti-imaging.

What is the reasonable for not use the mac as the identifier?

Trying to think of a use case I thought of wireless + wired, would be kind of neat if they had the same IP. But that falls apart completely if you have them connected at the same time (which I often do).

At one point I set up my wired and wireless interfaces as a bond with wired as the primary, and I could do a file transfer and watch the speed go up or down as I plugged and unplugged the wired interface. That was pretty slick.

That sounds cool. Did you document it anywhere for others to setup themselves?

We’re there any obvious issues in such a setup?

I did, but that blog was taken offline the beginning of this year when the company I was a part of when I wrote it went out off business. It was horribly out of pace with modern setups though, I did that ~20 years ago, it used a network manager that no longer exists.

I'm not sure how it'd fit into a Network Manager or systemd world. There really wasn't any particular trick to it, IIRC I used link monitoring to detect link failure, though I might have used ARP, and set the ethernet as the primary interface, just using the standard Linux bond driver.

With things like docking stations/usb-c docks/other adapters, the actual mac address doesn't necessarily identify a machine.

As my example illustrates, that is a feature.

Can't you just get rid of that file entirely? I would assume that there is a (sane) fallback for DHCP in case this file is missing? What other uses besides the phased updates and DHCP does that file solve?

Google Chrome reads it, probably for tracking purposes.

this is not an ubuntu only issue, happens on debian too

Reply on news.ycombinator.com