Hacker News Reader

Comment by LeoPanthera | Hacker News Reader

Best decision I ever made was setting up a Pi netboot server on my LAN. Now I have a whole bunch of Pi 3's and 4's scattered around my home with no local storage at all doing cool things, and I can make them boot into a completely different OS just by renaming a symlink on the server.

View on HN

> Now I have a whole bunch of Pi 3's and 4's scattered around my home with no local storage at all doing cool things

Elaborate? I have a single pi3 as a Plex server, looking for more ideas.

https://github.com/badaix/snapcast this seems really cool, although I haven't got around to doing it myself.

Snapcast is really cool, Johannes did an awesome job! I used it for a long time at home on multiple RaspberryPis. I've been working for some time on a more user-friendly and cross-OS alternative. It's available here: https://github.com/geekuillaume/soundsync

I think I'm going to try to integrate net-boot in Soundsync this weekend. It would be awesome to be able to install Soundsync on any computer on the network, turn on an option and then just start the rPi to get it to play music.

Two play music, one into my hifi, one into my home audio distribution system.

One is setup for RetroPie. One has LibreELEC installed, showing TV from my HDHomeRun. One has a GPS hat and is a local NTP server.

And one boots into an Amiga emulator because I'm very old and nostalgic.

(If you’re able to play to both simultaneously) Do the music playback devices stay in sync?

No they're independent. But if I want multiroom audio I just use my very low-tech home audio system. Most rooms have ceiling speakers that are all wired back to the garage where one of the Pi's lives.

I actually tried this and it turned out to be quite a bit more complicated than predicted, due to clocks on the pis being slightly different

"I'll just sling udp multicast of the PCM samples around the network, can't be that hard right?"

ended up using MPD with pulseaudio over TCP, and they get out of sync after a few minutes, but good enough

The different clocks is a real hurdle.

I have to give credit to Apple on this one, as they had the genius idea to make the buffer >1s, and send control messages independent of the audio stream.

There’s gotta be a way to do it with PTP and the same ideas but clearly it’s either an extremely difficult problem, too niche, or both.

What is the requirement for synched audio? milliseconds, microseconds, nanoseconds,...

I'm currently working on devices that synchronises themselves wirelessly indoor over a large area and I'm now wondering if music could be an application I haven't though about...

10 ms is fine. 100ms works but you get odd echoes like a speaker system in a big building.

You need some way to compensate for clock drift between different playback devices. Even good clocks will drift 10ms in a few hours.

1ms is barely "ok". with 1ms you'll still hear ugly comb filter effects. (see my other comments)

At most 10ms. Sub 1ms would be awesome.

And yes, definitely. If you can build in a “chromecast receiver” so that a user could cast to multiple synced speakers (at full quality), you’d definitely have my attention

It's worth noting that for two speakers playing the same audio, a 5ms offset will move the point of perfect sync by less than a meter. Going below a millisecond is definitely overkill territory.

In 1ms sound travels just over 30cms - so just moving around a reasonable room affects the offsets between two well spaced speakers by 5-10ms. I use my own multi room audio system at home - http://strobe.audio - and although the sync isn’t perfect by any means the effect of moving around your house usually the problem. Eg if you have the volume in another room loud enough that it’s louder than the music from the room you’re in you get the weird echo effect. Otherwise nothing.

Turns out our brains are really good at compensating for relatively large offsets

I beg to differ: a good rule of thumb is that an (audio) sample takes about 8mm (44.1k). By the time you're a full 1ms late, that's a 44 sample difference between the two (or that's as though the speakers were about a third a meter apart.

And that distance, you're gonna hear some uncomfortable comb filtering. A third of a meter corresponds to a frequency of about 1khz, which is right smack dab in the middle of a critical band, so you WILL hear the constructive and destructive interference around ~1khz.

(and _5ms_? That means the speakers are now ~220 samples out of sync, so I'd expect any lows around 200hz to have some destructive interference, making the music sound hollow.)

1ms of sync is FAR less than adequate. It took effort, but I was able to get a distributed set of raspis to within about 5 samples at 44.1k, and I think I can do better.

Why do I know this or care? :) I've done (and still do) a lot of research (and inventions) in the field of microphone array processing and audio source localization -- tracking objects purely by their sound. With cheap commodity equipment I can generally get a resolution of 5cm, and with lots of fine tuning I was able to do sub-cm tracking.

All those problems happen even if your speakers are perfectly in sync. These aren't headphones. If you have a couch for listening, the difference between sitting on the left and sitting on the right is just as big or bigger. You get the same interference, yet it isn't a problem.

I think his thinking is still valid.

If you sit on the left or right the sound should still come from "center stage" or (x,y,z)=(0,0,0) without being shifted or drift.

Depends on whether you have a stage. If you're playing audio to go with a TV you don't want it drifting much. But when the scenario is playing music over an area you end up with a bunch of mutually-contradictory centers and it doesn't matter if they shift over a little. And music over an area is what sparked this talk.

Correct. You use PTP+NTP, the RT_PREEMPT patches, and then you have to deal with a constantly varying resampling ratio (!!) And not only that, but if you don't want nasty artifacts, the a LOT of thought also has to go into your filter design. It's hazy, but I remember using a pair of orthonormal time-varying allpass IIR QMF filters, with extra black magic I'm blanking on that allowed me to reduce the complexity of finding the coefficients for, and computing the filters by at least a factor of 4. Fun stuff. I think half the coefficients in the EQMF were zero, and of the other half, half were the reciprocal of another, which led to a fast implementation.

But yeah! I've done that. Kept audio streams to within a few samples of alignment for more than 24 hours. It was fun, and yes, somewhat hard. "will more than happily do it again for money". :)

Why would you send samples? If you do any kind of compression you’re going to convert to frequencies anyway and it isn’t that much of a hassle to expand that.

That sounds like a better solution - I'd guess most a/v systems are used for pre-recorded video that could deal with more delay during stop/start/pause but giving you more synchronization.

To really do it right though I think it would be cool to tune everything somehow. So have a microphone + photodiode sensor you put in your chair, then run a test and have it synchronize all audio and video sources together. (maybe it could even be pointed to your bluetooth headset and do that delay too)

I think with that setup, you could not only get good synchronization, you could probably tune the delay to the minimal value (maybe you could still do stuff like something live)

Take a look at snapcast if you want to fix your out-of-syncness

https://github.com/badaix/snapcast

"Typically the deviation is below 0.2ms." Impressive!

I successfully used shareport-sync and forked-daapd back a while ago with PogoPlugs and USB audio interfaces for multiroom audio. It worked without too much effort.

https://github.com/mikebrady/shairport-sync

http://ejurgensen.github.io/forked-daapd/

do you have a blog about this, or willing to share any more info? This sounds like something I'd like to start working towards

I haven't documented it, but it's very simple.

I, too, would appreciate a brief Google Doc or Pastebin on this if you can spare the time. I've never setup netboot before, but have been interested for years... reply

Unless you are running your own router, like the op, most likely his setup won't help you. You should start by understanding all the components of the system and then figure out how to make it fit in your environment and maybe what changes you need to make. Start here [0]

0. https://en.wikipedia.org/wiki/Preboot_Execution_Environment

The Pi doesn't use PXE. It's different, but documented here:

https://www.raspberrypi.org/documentation/hardware/raspberry...

You probably should read that link, section "DHCP Request / Reply". Virtually the entire world uses PXE.

Anyway, I find it funny that we are arguing about the proper name for network boot in a thread about usb boot.

OK sure. :) I guess what I meant was, if you're setup for booting Pis, you can't also boot arbitrary PCs without extra work.

Not much extra work I’d have thought. You might need to hardcode the Pis MAC address in your preferred DCHP server but the rest of the config should be equivalent.

I’ve already got isc-dhcpd running with PXE booting for x86 / AMD64 machines so might add some rules in for the Pis too.

Thanks for the heads up by the way. I hadn’t realised you could network boot Raspberry Pis.

The issue is that the Pi does something that is almost-but-not-quite PXE. In particular, the RPI firmware does not send anything that identifies itself as an RPI so that the DHCP server can determine what packet to send in response -- the option 60 string it sends in the DISCOVER packet claims to the DHCP server that it is an X86 server running in legacy BIOS mode. The only way the RPI will boot is if you send it that extra magic option 43. For those of us that write DHCP servers that try to be intelligent about how to respond to PXE requests, the plethora of stupid not-exactly-PXE mechanisms the various ARM SBC hobbyist boards are annoying as fuck. At least the server class ARM stuff usually has a UEFI implementation that adheres to the UEFI extensions to the PXE spec and properly identifies itself with options 60, 93, and 94.

Can I ask what software you're using for music on your home audio system? I have an HTD system with speakers throughout and while Chromecast Audio was great for a while, stability has been an issue that I'm no longer interested in hassling with.

(Also looking forward to researching this netboot solution you speak of)

Do you know homeassistant?

https://github.com/home-assistant/plugin-audio

Yes. I have a lot of respect for the HA project, but I stopped using it last year. It changed far too much for me too much keep up at the time, and I ran into too many issues with the zwave support at the time.

Thanks for the link.

Not the parent but I have a similar setup. I use volumio music software on 4 raspberry pi 3b+, each with a hifiberry amp to drive speakers. If you couple these with snapcast, you have a diy whole house sonos-like setup. Works well and I control volumes through home assistant.

Not familiar with netbooting, but will be looking into it!

I had been considering a similar setup and then got a notification for a barely used htd lync12 with amps on Craigslist at an insanely good price and hopped right on it. I've spent the past year or so wiring up the house a room at a time, which has been painful and sometimes stressful. But I've become such a huge fan of the simplicity and stability of a wired setup that it's all been absolutely worth it.

Thanks for the recommendation. I'll give volumio a closer look.

I use Moode Audio. https://moodeaudio.org

This looks great. Thank you

Also not the parent. For me, I want my speaker, position 4 meters away, to play some music from my phone or laptop. First, I tried to use use youtube feature to beam the video to my android smart tv connected to the speaker. However, it is a waste of resources to do that. Later, I found some scripts on the internet for my Pi 3 to play the sound via Bluetooth. It is a better solution than my original idea, I can use any app, not just youtube. However I recently found an even better way, by buying a Bluetooth transmitter dongle that can be connected to my speaker. It only costs me less than $2. I can ask my Pi 3 to do something else more important than this task.

Surprised not to see Pi Hole on that list!

Any old SBC can run that reliably on it's own, the parent is talking about swapping out the OS with netboot on the fly.

My router runs pfSense so I don't need Pi Hole, I can use pfBlockerNG instead, which is built in.

Emby also offers the ability to control and play stuff on other Emby devices on the same network. LibreELEC/Kodi + the Emby plugin [0] on a Pi makes it appear as an Emby device on other devices. Been using that for more than a year an it works like a charm. Plus the data is fetched from the Emby server and not device to device, so you can play something from your phone to your TV, then continue to control from your tablet while your phone is off.

[0] http://kodi.emby.media/

How do you serve up images to the retropie box?

Not related to the rest of the thread, but this popped up the other week and, honestly to my shock, worked well on my Pi3.

https://news.ycombinator.com/item?id=24441112

You may or may not get some latency with video. Seemed to vary for me.

I thought this was going to be about NymphCast, an open source alternative to chromecast

http://nyanko.ws/product_nymphcast.php

https://github.com/MayaPosch/NymphCast

Oh, yes!

I should have remembered this one, since I am particularly interested in it.

But, I have not actually tried it myself. I wish I had the immediate chops to contribute to the project.

pihole works really well for me as a LAN-wide ad blocker.

Does anybody know of something that will serve up tracker and game music files (e.g. .mod, .xm, .it, and .spc, .nsf, etc.)? I'd really love to be able to use one of my Pis as a compact music server with a web interface that will on-the-fly transcode and stream these old formats. I have literally tens of thousands of these sorts of files and outside of converting them all to TBs of mp3s wouldn't mind at all to serve these up from their smaller original formats.

MOC will play mods as well as other audio, and since it can be controlled over the console, you can probably hook it up to a web page too, if you're prepared to tinker with it.

mopidy/mpd

Woah, this might be it. Thanks!

>looking for more ideas

1. MotionEye can help set your own local security camera setup.

2. PiHole with Unbound can help setup network wide ad blocking and DNS over TLS.

3. SDR (DVB-T dongle) with rtl_433 can receive 433Mhz communication from your sensors like Fire Alarm, Gas Sensor, Car key etc. and notify you.

4. MQTT server on a Pi can act as a local notification server, MQTT client on your phone can receive notifications.

Of course, the possibilities with a SBC are endless, but I included just those projects which I feel can be implemented without much effort but provides huge lifestyle/cost-benefit improvements.

On 4, what MQTT broker(s) would you recommend? I’be been reading up a bit on different alternatives[0] for something lightweight with not too many dependencies to run clustered/HA and haven’t figured out which one to try yet.

[0] https://github.com/hobbyquaker/awesome-mqtt

Mosquitto seems to be the de-facto "standard" choice for a Pi. Easy to set up and definitely lightweight.

Yes, Mosquitto from default repo works as it should without any trouble.

As for android client, I recommend this[1] but it requires Tasker Autonotification plugin to display notifications.

[1]https://play.google.com/store/apps/details?id=in.dc297.mqttc...

Can you host the netboot server from a workstation by chance? This is something I had never considered and it would really change things up for me, how does that even work? Is it just an installed distro or just a network image on the device it's booting from? I'm somewhat familiar with netbooting an installer but I've never setup a server just used the tech to install an OS from the network to hardware.

I did bump into these docs:

https://www.raspberrypi.org/documentation/hardware/raspberry...

Are these the gist of it, or did you do a little more?

Well, that's instructions on how to setup a Pi itself to be the netboot server. That will work fine, but your DHCP server still needs to be configurable to emit the correct responses to find the TFTP server, if you don't want to use a Pi itself as your DHCP server, which I would imagine most people don't.

My DHCP server is pfSense, and I'm using a FreeNAS server as the TFTP/NFS server.

The Pi netboot procedure is documented in more general terms, here: https://www.raspberrypi.org/documentation/hardware/raspberry...

Pi notwithstanding, it feels like this process hasn’t changed in 30 years.

I know for a start Microsoft made some ‘updates to the standards’ when they found out tftp isn’t really suitable for fetching gigabyte size images.

> That will work fine, but your DHCP server still needs to be configurable to emit the correct responses to find the TFTP server, if you don't want to use a Pi itself as your DHCP server, which I would imagine most people don't.

Why wouldn'y you? That's what I do: I use the DHCP server (dnsmasq) that comes with pi-hole, with a few additional config files, and it works pretty well. It even grants leases faster than my router was (which has been an issue for radv packets & DNS).

How do you handle persistent storage? I find it is often non-trivial to adapt an appliation-specific operating system to run via pxe boot.

No one in the thread has mentioned it yet, but NBD is also an option, either NBD root or NBD for particular filesystems. I've used it for managing Pis and a couple of SiFive boards. It has different trade-offs from NFS (not necessarily faster/slower/better/worse, depends on what you're doing).

For what I'm doing which is software builds I was able to write a specific NBD driver[1] for this task which has quite large performance gains, although there are trade-offs that only make sense for temporary build directories. I don't believe this is possible with NFS at all, or at least not without far more effort.

[1] https://rwmj.wordpress.com/2020/03/21/new-nbdkit-remote-tmpf...

Probably the most important tradeoff between nbd/nfs is that NFS by virtue of supporting shared mutation, means client-side caching is either non-existent or barely useful

With NBD you're giving a filesystem driver exclusive access to a block device, the kernel knows the underlying isn't expected to change, so for example no round-trips are required to read or revalidate some cached data, and something like "find /" while slow to run the first time around, will likely be served from cache on a subsequent pass, and thus lightning fast

This is very true, but I think NFSv4 makes things a little better?

What are you using for a filesystem? NFS?

My worry would be saturating the older 100Mbit connections when downloading things like docker images.

That's more a problem of docker than NFS though. You probably would be better if you just use it traditionally or use something like NixOS.

There was a project that I read about a while ago: https://matthewbauer.us/blog/nixiosk.html

The Pi4 can do 300-350mbps, which is faster than most people's internet connection.

Netbooting and mounting NFS root would be done over local lan, the speed of internet connection has no relevance there.

Right, I was referring to my parent comment that talked about pulling Docker images which come from the internet unless you set up a proxy.

Ok, that depends on the docker images; things like pihole are relatively small (300 MB, including all layers); I would be more afraid of running containers over NFS slowed down by rpi ethernet, than downloading the images themselves.

Agreed. I'd have it set up so that there is some local storage via a USB3 thumb drive to hold the container images, which anecdotally pretty easily hits 80MB/s for reads, often hovering in the 100-110MB/s range.

True, but if you're netbooting you're probably not going to the internet.

Actually, just tested on my local network, got ~940 Mbps sending and receiving. Pi4 has real Gbit now, not limited by the USB bandwidth any more.

I can download large files from my pi at 900mb/s and upload around 800mb/s. The hard drives attached to the pie are formatted ext3. If you use an NTFS drive though, you will get speeds around what you are describing.

Yes, TFTP for the boot rom, and then NFS for the root fs.

I don't use docker or any other kind of container system because it's so easy to just reboot a Pi into a new OS on demand. Everything runs natively.

That's what we did when developing our original Embedded Linux stuff. Once you get it working it's really nice. You have your standard tools, version control, etc running locally on your system.

100mbit is similar to a slow sd card or usb stick anyway?

> My worry would be saturating the older 100Mbit connections when downloading things like docker images.

Is that too slow? I feel like 30 seconds to fill up half the ram is a perfectly acceptable boot time.

I did this, but using a more traditional server. It was... OK. The pi3b+ bootloader was flaky and I ended up needing an SD card with just the kernel on it, but I wanted it ro. This made updating kind of touchy.

I can imagine that if they've made the bootloader more robust on the pi4 that this is truly excellent now.

If you already have the SD card and USB, your probably better off using https://github.com/pftf/RPi3/releases/tag/v1.30 which is a open source UEFI implementation. Its far more capable of dealing with underlying os updates using your OS's preferred update method. You put the UEFI firmware on the SD, update the config.txt to point at it, and then pretend the rpi is a normal x86/uefi system and use apt/zypper/dnf/whatever for updates (including the kernel). Everything then boots/runs/updates off the USB disk.

The arm ecosystem seems to be slowly moving to some form of UEFI+grub boot for all the linux/etc distributions and away from all these platform specific boot methods.

That requires your network attached storage (NFS/iSCSI or something else) to be HA ;-)

I started booting from USB SSD a couple of months ago since I got my Pi 8GB, intended to use it as my low power consumption workstation replacing the Optiplex SFF Running Manjaro + KDE Plasma. Turn out to be, Manjaro ARM + KDE Plasma on Pi 4 runs pretty well (even all KWin eye candies work as well as the Intel HD integrated GPU on the x86_64 variant). However, USB 3.0 to SATA III SSD is clearly a bottleneck still. I am yet to check if NFS over 1Gbps ethernet can be faster (anyway, my btrfs backed consumer grade NFS may not be production grade to support 10+ devices).

What happens when you get a power cut?

Or, assuming you are from a well functioning country, what would happen and how would you restart?

I'm not sure I see the problem. Most servers will come up before a netboot/PXE boot timeout. At least in my experience they do.

That server should be on a UPS. Some people like to use an old laptop as a server at home, and that's even more ideal as it has a built in battery, often allowing the machine to stay alive for an hour or so before dying.

Thanks, this answers my question. In some developing countries you could have power cuts of over 10 hours.

> just by renaming a symlink on the server.

Wow. How?

Netboot instructions are here: https://www.raspberrypi.org/documentation/hardware/raspberry...

The important bit is that it will look for its boot files in a subdirectory named the serial number of the Pi. Only if that fails will it look in the root.

So I have symlinks for the serial numbers of all my Pis that point to whatever OS I want to boot.

Very interesting it looks for the serial number and not the MAC address!

Pretty sure that's because the pi doesn't actually have a hardcoded MAC; I think I've read that it's just set in software and persisted by the OS, which is weird but workable right up until there is no local storage to persist on.

Thats awesome. I asked about people netbooting for retropie in reddit and people were really confused why I would want to do that. I think I'm going to go ahead and set it up now.

Whaty the best way to do this without trusting the untrusted LAN?

If you just want to prevent other users from snooping you, use VLAN.

There's no protection against the network administrators though.

Pantera, é você? :-O

what hardware does the netboot server run on?

It's a Dell Poweredge running FreeNAS, which I was already using as, well, a NAS.

The required DHCP responses come from my router running pfSense, which is some kind of cheap Supermicro thing.

Reply on news.ycombinator.com