Hardware hotplug events on Linux, the gory details | Hacker News Reader

Hardware hotplug events on Linux, the gory details

arcanenibble.github.io

172 pointstodsacerdoti20 comments

Sort of relatedly: I bought an inexpensive DDR pad recently that worked on coldplug but not hotplug.

It turns out the firmware on the pad isn’t quite ready to be polled for USB descriptors right when it is plugged in so you have to put in a little udev hack to suspend it then let it reconnect. At which point it comes back correctly.

(The manufacturer included a little debugging flyer telling you to plug the device in slowly to work around this issue haha)

For those interested: https://github.com/batocera-linux/batocera.linux/issues/1547...

> we can see that there are two hotplug backends: linux_netink.c and linux_udev.c

I can't help but think that netink would have been a much cooler sounding word than "blog".

… mixing endianness within a packet is diabolical.

There's also netten, the network interface and protocol in which time reversal is entirely normal and used for many cool tricks.

A while back I built an orchestrator application that would monitor usb events and then launch/destroy containers that have that usb device mapped to them. The orchestrator also ran in a container with no networking.

Imagine my frustration when I learned that udev events don’t make it into containers unless networking is enabled.

Figured out a solution eventually. Was a combination of compiling certain packages from source and some kernel flags if i remember correctly.

I can't help feeling that the old XKCD cartoon [1] about life satisfaction being proportional to the time since last opening xorg.conf could equally apply to udev.

For instance, I tinker with FPGA boards, and one board in particular presents both a JTAG and serial port over USB. Nothing unusual there, but while most such boards show up as /dev/ttyUSBn, but this one shows up as /dev/ttyACM0. I eventually figured out how to make the JTAG part accessible to the tools I was using, without having to be root, via a udev rule. The serial side was defeating me though - it turned out some kind of modem manager service was messing with the port, and needed to be disabled. OK, job done?

Nope.

A few days ago I updated the tools, and now access as a regular user wasn't working any more! It turns out the new version of one particular tool uses libusb, while the old version used rawhid (that last detail is no doubt why I had such trouble getting it to work in the first place) - and as such they require different entries in the udev rule. I'm getting too old for those kinds of side quest, especially now a certain search engine is much less use in solving them.

(Not naming the tools because I'm not ranting against them - just venting about the frustration caused by the excessive and seemingly opaque complexity. Having got that off my chest, I'll go read the article, in the hope that the complexity becomes a little less opaque!)

[1] https://xkcd.com/963/

> it turned out some kind of modem manager service was messing with the port, and needed to be disabled.

Curious. What service was that?

I have an on-board serial port that's only working in one direction, which is something I've never encountered before. I wonder if the service you're referring to could be causing my problem.

ModemManager. You need to set the variable ENV{ID_MM_PORT_IGNORE}=“1” I. A udev rule.

Standard usb serial ports show up as ttyACM#, whereas nonstandard ports that require a driver like ftdi show up as ttyUSB#. Modems tend to be standard usb devices, so ModemManager by default scans all serial ports as if they were modems. This involves sending some AT commands to them to try and identify them.

Software implementations of serial devices tend to follow the standard, so they show up as ttyACM#.

Thanks for the tip. Unfortunately, it doesn't seem to be the cause of my one-way serial port issue. Adding the udev environment variable makes no difference, nor does stopping the ModemManager service.

ModemManager used to open() and probe every tty device attached to the system. I had a 8-channel relay card with an arduino nano wired up with my desk to control the lights and disco ball, interfaced with a custom ascii-based serial protocol. connecting it to an ubuntu machine (where modemmanager was active in the default install) turned the 2nd or 3rd channel on.

This was generally infuriating, there are many arduino forum posts about modemmanager messing up DIY setups.

Upstream fix was changing modemmanager to work on a whitelist / opt-in approach instead of blacklist / out-opt. My fix was to switch to debian.

These rules exist ostensibly for security, right? I wonder what the right model is here for interactive end-user operating systems. Just trust apps to behave and give them access to your devices? That's more-or-less what udev hacks end up amounting to in my experience... Maybe the API applications see should just ask the OS for a device that matches some description, and then the OS pops open a picker for the user, kinda like a file dialog? Selection of a device to pass to an application counts as granting permission to use it.

Devices are typically controlled per-user. So if you want to increase isolation you need to run as a different UID. It's definitely a sub-par model IMO but it seems to work well enough.

> and then the OS pops open a picker for the user, kinda like a file dialog?

How would that work when I'm in a container or at a tty with nothing more than a shell?

> How would that work when I'm in a container or at a tty with nothing more than a shell?

I only really am considering designing for graphical systems. If you're doing server work or devops configuration living in a udev rule file feels more reasonable.

I don't have a problem with the concept of udev - I just find the details to be laden with papercuts. Needing different syntax depending on which subsystem is interacting with the device makes no sense from a user perspective.

To cover all bases my udev rule seems to need to contain both

  SUBSYSTEM="hidraw", ATTRS{interface}=="CMSIS-DAP", MODE="0660", GROUP="plugdev"

and

  SUBSYSTEM="usb", ATTR{idVendor}=="1d50", ATTRS{idProduct}=="602b", MODE="0660", GROUP="plugdev"

I don't understand why the second rule isn't sufficient, and finding enough information online to even consider trying the first rule was extremely difficult.

I think the main problem is there are several layers of events for the subsystems and absolutely no documentation on how they work!

More to the point, no documentation on how they work this year.

Shouldn't you manage permissions from groups instead of hacking udev?

And why do you have modemmanager if you don't have a modem?

> Shouldn't you manage permissions from groups instead of hacking udev?

Well my user is a member of plugdev, but by default udev has no clue that it should allow plugdev members to access some obscure third-party FPGA board. Someone has to write a udev rule for it, and if they don't share it for others to use, so does the next person. The next person happened to be me.

> And why do you have modemmanager if you don't have a modem?

And that is the right kind of question! I have absolutely no idea why my stock install of Linux Mint includes and activates ModemManager.

Love it.

Since I am a visual learner, here is a sequence diagram that helped me follow it a bit cleaner. (yes, I used the gAI dark magic)

```

  sequenceDiagram
      participant HW as Hardware
      participant Kernel as Linux Kernel<br>(USB / driver core / kobject)
      participant NetlinkK as Netlink<br>(NETLINK_KOBJECT_UEVENT<br>group 1 = MONITOR_GROUP_KERNEL)
      participant Udevd as udevd<br>(systemd-udevd)
      participant NetlinkU as Netlink<br>(NETLINK_KOBJECT_UEVENT<br>group 2 = MONITOR_GROUP_UDEV)
      participant App as Userspace Application<br>(libudev or direct netlink listener)
      participant Sysd as systemd<br>(device units, services)
      participant DevFS as /dev<br>(device nodes + symlinks)
  
      HW->>Kernel: Physical insertion (USB plug-in)
  
      Kernel->>Kernel: Detect change via bus/driver<br>(e.g. xhci-hcd → usbcore)
  
      Kernel->>Kernel: Register new device in device model<br>(kobject_add / device_add)
  
      Kernel->>NetlinkK: kobject_uevent_env(ACTION=add, ...)<br>multicast to group 1<br>(raw uevent: null-terminated key=value strings)
  
      NetlinkK->>Udevd: Receive kernel uevent<br>(ACTION=add, SUBSYSTEM=..., DEVPATH=..., etc.)
  
      Note over Udevd: udevd parses uevent
  
      Udevd->>Udevd: Match & apply udev rules<br>(/lib/udev/rules.d/, /etc/udev/rules.d/)
  
      Udevd->>Udevd: Perform actions:<br>• Load firmware<br>• usb_modeswitch<br>• Set permissions<br>• Run programs/scripts
  
      Udevd->>Udevd: Create device node(s)<br>e.g. /dev/bus/usb/001/002
  
      Udevd->>Udevd: Create symlinks<br>e.g. /dev/ttyACM0, /dev/disk/by-id/...
  
      alt Optional: triggers systemd .device unit
          Udevd->>Sysd: Triggers / influences device unit activation
          Sysd->>Sysd: May start dependent services / scopes
      end
  
      Udevd->>Udevd: Build enhanced udev packet:<br>• libudev header ("libudev\0", magic 0xfeedcafe, ...)<br>• MurmurHash2 subsystem/devtype<br>•   64-bit tag Bloom filter<br>• Original + added properties
  
      Udevd->>NetlinkU: Broadcast processed event<br>multicast to group 2<br>(binary format with header + properties)
  
      NetlinkU->>App: Receive udev event packet<br>(via libudev_monitor or raw netlink socket)
  
      App->>App: Parse header, validate magic/credentials<br>Extract properties
  
      App->>App: React to device<br>(open /dev/..., query sysfs, etc.)
  
      Note over DevFS: Device now usable via stable names / permissions

```

sorry for the dumb question, but what language is this? dotty? mermaid? what tool can i feed that code into ?

EDIT: chatgpt correctly identified it as mermaid.

live link: https://mermaid.live/edit#pako:eNqVVu1u6kYQfZWRf1SJLmAgJASri...

apologies. Yes it is for mermaid.live

Reply on news.ycombinator.com