Going in circles without a real-time clock
rachelbythebay.com
rachelbythebay.com
WireGuard only requires a monotonic clock, because it periodically rotates keys to provide the forward secrecy. Peer clocks are otherwise not required to be synchronized [1]. I guess the clock had a higher rate than usual and it couldn't be corrected due to the lack of RTC?
[1] https://www.wireguard.com/papers/wireguard.pdf#page=7 "In fact, it does not even have to be an accurate timestamp; it simply must be a per-peer monotonically increasing 96-bit number."
By the time you buy everything you need to make a Pi 5 even remotely usable and reliable today, you'll have spent as much as a small form factor PC. What do you need?
1. Special power supply - Yes, it's USB-C but it requires a high amperage 5V supply instead of accepting a higher voltage like most USB-C stuff
2. Active cooler - Pi 5 throttles immediately without it; it's not an optional component
3. NVMe Expansion HAT - Seriously why the fiddly little PCIe header? Put an NVMe slot on the bottom and let people adapt *that* connector
4. RTC Battery - At least put a damn capacitor on the board
5. Enclosure - I'm halfway expecting the pi foundation to print some dashed cut lines on the cardboard box so they can say it comes with a case =]
Not like the Pi Foundation couldn't have contributed to uboot or the upstream kernel or anything before thier hardware was released. Pretty much nothing released for "Raspberry Pi" runs on a Pi 5 at this point in time.
They better try to salvage something before putting out the expected CM5, but I'm not expecting much.
They could have just use any old less powerful chip, like, whatever has at least 3B performance is fine, and used the money saved for stuff we actually need.
Maybe ditch those awful micro HDMIs, it's kind of a nasty connector and USB C exists.
Onboard lithium charging is a few cents. Better yet, onboard LTO charging, but it seems nobody does that yet. Why just stop at an RTC battery when you could have a few minutes of real backup power to help stop SD corruption?
While you're at it, add an onboard I2C FRAM or battery backed SRAM.
Instead of NVMe why not hire a few devs to go around fixing stuff that can't run well from the SD card?
I don't know how you reach that conclusion; the network connectivity of pi's suck (especially pi5) and out of the box they do not support any traditional storage except for a couple of external usb drives.
The use as a low cost desktop is a primary mission statement of theirs, so I don't really read that part as a criticism though. I think it's probably the only thing it really does well.
Agree on your other points though. Other vendors have made lots of headway in the SBC markets in the years that Pi hasn't been able to ship product. I'm sorry in a way that they have not achieved their lofty goals, but in another way I'm not sorry because they really have been fucking it up bad for the last few years.
But I agree that the price to get a happy Pi is pretty harsh compared to a tiny PC.
Apparently, cable quality is also demonstrating itself to be a significant factor in the Pi 5 power supply environment.
Regarding both these points:
Then how the hell does it negotiate the power and know if it's attached to 3A or 5A?
The "Official Pi 5 USB-C Power Supply" is USB-PD compliant and as it is advertised as such in conjunction with the Pi5, I think that is where the confusion originates
The things I've been searching in the last few minutes suggest that the Pi can negotiate 5A but only if the power supply explicitly offers it as a PDO, which almost nothing does.
But even if that's right, it's almost as bad as not doing PD at all.
Edit: The documentation states "If the Raspberry Pi 5 firmware detects a supported 5A-capable supply, it increases the USB current limit for peripherals to 1.6A, providing 5W of extra power for downstream USB devices, and 5W of extra onboard power budget."
It also confirms no PPS, but it's not entirely dumb.
Here's the thing though; the simple fact that we have to even have this discussion because the Rpi5 continues to be super damn weird is enough evidence for me that the platform has jumped the shark. I don't know whether to blame Broadcom or the Pi Foundation, but the Pi 5 suuuuuucks. Man, I wanted to love it though; it's the only board I have even been been able to buy from them in the last FOUR YEARS
I find it hard to believe that it can be worse than RPi 3, which could try to draw up to 15W => 3A under the highest loads, while USB 2 could officially only go up to 2.5 W = 0.5A at 5V, resulting in countless corrupted SD cards. (15 W being the minimum maximum for USB C.)
The tooling for light software development and hobby embedded hardware development has been great so far.
Previous discussion on similar thing: "Encrypted DNS + NTP = Deadlock" https://news.ycombinator.com/item?id=34177331
Background: I use an x86 box as home router. I've changed the configuration in the BIOS that it should automatically boot up on power. However if the BIOS battery is dead, the config will be lost and it will revert to default settings, which is not to boot on power.
But there is no way to know how much juice is left in the BIOS battery, thus there is no way to issue warnings that the battery should be replaced soon. If there is power loss AND the battery is dead, when the power comes back up again, the router will just stay off indefinitely until it's manually turned on.
That's when I realized why most consumer routers do not feature RTC nor do they require batteries.
True, but its voltage can be used as an estimate. Most PC MB still around monitor the supply voltages, some also of the battery feeding the RTC. If you have the (Debian) package 'lm-sensors' installed, look for
sensors[<pid>]: Vbat: +2.91 V (min = +2.70 V, max = +3.63 V) ALARM
https://hwmon.wiki.kernel.org/lm_sensors also has a FAQ.
You should do it every year or two...
Maybe I'm just lucky.
It would be nice if DHCP could provide the current time as one of the pieces of information it provides with a lease. If you’re already trusting it for your IP address, you might as well trust it with the current UTC time: The current 64-bit timestamp would be enough to get timestamp-sensitive things working.
I don't follow! How does trusting DHCP with IP address automatically mean I should trust it with the current UTC time?
Time requires higher degree of trust than IP.
I may not care what my IP address looks like but I might care a lot about what the current UTC time is and I might want this to come from a more trustworthy device.
The DHCP server provided value can not be much worse than 1970-01-01T00:00:00Z default value if you don't have any other data. And if you have some other data, e.g. ext4 superblock timestamps, you can pretty trivially protect against DHCP providing time from the past (i.e. use the maximum of different sources).
Finally, you can restrict the use of the dhcp provided time to the initial bootstrapping process only; it's not necessary to use it for system-wide clock
In contrast, a different IP than expected isn't such a big deal... Although it might break the collaboration of two computers as crude denial of service
DHCP can provide the IP address of one or more NTP servers to consult for time queries. I use this feature so that all of the machines on my LAN (at least the ones that can be configured to ask for and/or use this information) have the same general idea of what time it is.
[0] See subsection 8.3 here: <https://www.rfc-editor.org/rfc/rfc2132#section-8>
That said, if you don't trust the DHCP server because some evil actor might give you the wrong time, why should you trust the NTP server it tells you about?
That goes without saying.
I mentioned the fact that DHCP can provide NTP server information as a way to explain why DHCP wouldn't just straight-up provide "current time" information. Why do that when you can point people to a server running a far superior purpose-built timekeeping protocol? It'd just be silly to do the worse thing.
(Notice also that RFC2132 was published in 1997. In my mind, this moots any retorts that a good reason for providing time directly would be because the DHCP server's supplicant is too underpowered to run an NTP client. In 2024? On a consumer-grade device? No way.)
I don’t know how widely this DHCP RFC is supported though.
Anyway, I solved these problems with sbts-aru, but maybe it's interesting for you to test if you haven't already done this.
refclock pps ppspath /dev/gpspps0 prefer time1 0.004 minpoll 3 maxpoll 4 iburst
refclock nmea baud 57600 time2 0.068 minpoll 3 maxpoll 4 prefer iburst
NTPD_OPTS="-ggg -N"
When I switched the startup of gpsmon and the chrony daemon to the other way around time sync was achieved within 1/2 minute to a minute (Real fast) and the reported time on two separate Pi's with the joined up gpio's (So you can get very close to triggering a time check at the same time) was then sometimes as low as 30 ns's from each other, which is damn good considering that most of the difference would be jitter.
And none of the how-to's that I read talked about changing the start order of chronyd and gpsmon so I can only assume they all might be suffering from the same problem. It's important to note that the diagnostic commands from chrony will not indicate this and the only way you will notice if this is happening if you setup a rig like I mentioned above. I noticed because my sound localization calculations were showing errors that didn't make sense.
https://shop.raspberryshake.org/product/turnkey-iot-atmosphe...
Currently I only support USB mics at 16 bits that can talk to jackd. This was a conscious decision because it was all that was needed but also because jacks makes it essentially have multiple source sinks. Such as as well as recording a real time audio processing pipeline that potentially leads to real time gunshot and other sound event localization.
But it would be cool if the infrasound sensor had a USB sound interface. As then it could be useful in localizing elephants.
PPS turned up quickly on Wikipedia, so that one is readily answered
> PPS signals have an accuracy ranging from 12 picoseconds to a few microseconds per second, or 2.0 nanoseconds to a few milliseconds per day based on the resolution and accuracy of the device generating the signal.
* Clayface: https://www.developerdan.com/ntp/#./clayface/7-days/
* Catwoman: https://www.developerdan.com/ntp/#./catwoman/7-days/
That Wikipedia quote should mention temperature! Temperature variations have a big impact at this level of accuracy. These really cheap GPS receivers do not have temperature adjusted clocks. Unfortunately my server closet (this is just a hobby) does not have well regulated temperature, so you can see the impact of temperature on the clock accuracy. Also, I found if I start running a bunch of stuff on these computers - that makes the CPU heat up, which also affects the jitter. If you really want high-precision, you'll have to shell out some extra cash then I did: https://www.sparkfun.com/products/18774
I'm going to have to check this out! Thank you for sharing!
I only ever use USB storage on my pi's now and not had a single issue since
I've always been amazed that the Foundation never mandated the use of validated brands or types of SD cards.
Not that it matters much to me, I've had to retire all of my Pis earlier than the 4. It seems that all of my Raspberry Pis prior to the 4 all had some sort of design issue with their power regulation. After a couple years of constant operation, they just start locking up randomly around once every day or so. I've had everything from a Pi 1 through 3 exhibit this behavior. I spent WEEKS troubleshooting this and conclusively ruled out wonky SD cards, power supplies, connected devices, proximity to other equipment, and every other factor I can think of. I found many threads with other people reporting the same issues but the Foundation refuses to acknowledge it as a problem. I only run Pi 4's now, and am gradually switching over to Intel N100-based systems for low-power operation.
Using an SD card is just nuts.
For those of you who didn't read the fine article, I've quoted it here:
> As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.
Since clients will attempt to resolve ntp.org in order to actually sync their clock, there is a good probability that some clients will be way off.
Enabling dnssec on that zone was probably not without important drawbacks? I wonder if the operators thought about that potential pitfall. Seems like they might be doing a disservice to their core mission of allowing devices to sync their clock.
Is my clock more likely to be accurate after NTP.org got signed? It looks like it's less likely, on average.
If memory serves though, Raspbian used to not even have `fake-hwclock` by default and even more Pis would end up with this "wildly wrong time near the epoch and can't bootstrap DNS/NTP" failure mode. You'd sometimes also see it with VMs too (esp if they were doing a pure-software clock that ran slower than realtime, instead of patching clock calls to the real hardware clock on the host).
Preventing DNS mitm only matters if you prevent NTP mitm too.
What percent of NTP clients talking to these servers are doing it in a secure way? And is that share growing?
Roughtime would be far better, but essentially there's no broad deployment of it yet.
Ideally something good would be picked by Raspbian and delivered in the distro as standard.
For a lot of micro-controller timing uses, they aren't required; for example, hardware timers based on the primary clock source, or the Cortex-M systick, but for maintaining accurate dates and times over long periods, the RTC is the right tool. It can also output the dates and times in a convenient format as well, as long as you find named register fields convenient!
I use an external RTC with a Raspberry Pi for some sensor systems I work with. Its purpose is to give a reasonable chance of having the correct time for recording data in remote locations where no internet connection is available at startup. For a Pi that only provides network services (and thus always needs a working internet connection), I probably wouldn't think to bother.
That was actually the easier hardware platform, because it either had good NTP sync or it didn't reach our C&C servers.
The difficult one was the platform that had an RTC. Those would slooowly drift out of sync if NTP was broken. Slowly and silently.
https://github.com/hcfman/sbts-aru
Problem solved.
There's no reason now for all geeks not to have at least one stratum #1 time server in their networks. Really none.
Local stratum 1 GPS time (even with PPS pin) servers based on Pi's can suffer similar issues when they lose satellite lock (weather etc.)
Have a great day =)
Why all the complex solutions when it's so easy and cheap now to install a stratum #1 time server, even a Pi 3 will suffice. It works on a Pi Zero, but then you only have a wifi link.
The sbts-au project make's it incredibly easy to setup. git lone and then install. One command, it's that simple.
EDIT: Any any advice on how essential PPS is?
Some U-Blox modules support the additional ground based navigation signal and cellular beacons. This is recommended in addition to the 3 satellite networks classes, but note some modules will not work with some antennas (get the unified hardware with built in antenna/lna if unsure about RF notes).
Best regards, =)
Adafruit ones also work great, but are a lot mor expensive, I do have several though.
PPS is criticial to proper timing. However... I was using the time synched systems for sound localization. But as they only cost around 7 euros on aliexpress, why not.
systemctl enable systemd-time-wait-sync.service
mkdir -p /etc/systemd/system/wg-quick@wg0.service.d/
echo "[Unit]
After=time-sync.target
Wants=time-sync.target
" > /etc/systemd/system/wg-quick@wg0.service.d/override.conf
Then the real solution might involve ensuring good clock sync. Using a GPS synched time source is good for this, such as this one :) https://github.com/hcfman/sbts-aru
For some people, all their network access goes over wireguard and that includes time syncing. That's not the case for me, but for those people if wireguard isn't working their clock can't sync.
That GPS receiver looks cool! However an RTC and a battery will do just fine for me, which is the plan.
The author is allowed to ask you to shut up and go away if you do.
Sharing cautionary stories is good. Learning from others' mistakes is good. Jumping on these as an opportunity to put someone down for the purpose of self-promotion ("mostly self-inflicted", express your surprise) is unkind, probably bullshit (I have seen soooo many people pretend to be above problems and then run smack into them), but most importantly: it shits up the dynamic where people are willing to share their mistakes, allowing group-learning. It would have been the easiest thing in the world for rachelbythebay to not make this blogpost and thereby avoid providing an opening for asshats like yourself, but I'm glad she made it anyway, even knowing that you would take your shot, so that everyone else could learn.
I guess I need requests to the NTP server to go over WAN directly. Always seemed like a hassle to get this to work with openwrt zones and stuff. My eternal gratitude to anyone who shoves me in the right direction...
— Restoring more sane shutdown timestamp from file instead of just using 1980.
— Making blind single use ntpdate request to some public server to set local time after the network is available, but before any other application uses it. There might be appropriate hooks in your init system if you don't want to make it an independent step.
— The same, but using some local time source. It is not widely known that even plain old Windows desktops, while not having a complete NTP server, can be trivially configured to announce time over some form of SNTP. If you have any computer with a real clock running nearby, you can probably rely on it. Next step router is also a good potential candidate.
As for compartments and access control, you can try to use basic VLANs (one for ntp traffic, and only it, and one for VPN) or maybe some two-step network configuration process (get time with one configuration, reset and proceed with other services).
All of that is not a problem, the problem is what to do when something fails. Time server becomes unavailable, intermittent internet connectivity, server you use for pings is available, but others are not, etc. You can choose to stop completely if there is no time source, and wait, but that might render the system unavailable for remote configuration. You can try to restart the device or restart the network router automatically (though only a fixed number of times, restart loop will surely break something), but that won't help if the problem gets fixed some time later. You can also use a fallback configuration that doesn't depend on correct time (e. g. only start an ssh server so you can connect to it).
Do you have something on the LAN side that could run an NTP server? I'm thinking you could add your laptop to your Pi's list of NTP servers, then just start that service up temporarily as-needed to give the Pi a time that's good enough, then it can get actual correct time sync from somewhere else once Wireguard is working.
I’m actually setting up a router with a rpi4 right now, only using Fedora IoT instead of OpenWrt. It’s a bit more assembly required lol
It’s definitely served me well in a wide variety of disciplines.
Multi-booting PCs with Windows and Linux over the years, I have seen the time sync problem go from nonexistent to show-stopper.
For devices that you only need to connect to the internet occasionally or sporadically (so that's what you do), that's where I noticed it most.
Linux sets the RTC to UTC, then when you reboot to Windows, Windows uses the RTC as local time. Which for me is 5 or 6 hours different from UTC, depending on Daylight Savings Time.
Plus when Daylight Savings comes around, each OS wants to make a 1 hour correction the first time you boot it after that date. With multi-booting this can add up to more than one hour difference too.
Didn't used to be so bad, if you were a few hours (days or longer for many PCs) away from the actual time, things did not fail for this reason.
Eventually only one hour off was OK for a while, now nope.
There are just so many more obstacles to smooth reliable operation, and weak links in a more extensive chain that must remain perfectly strong. Otherwise the chain is broken, you are disconnected, and the weak link is too many sections away from you now to be within reach.
Complete perfection is required more so than ever, while at the same time, your efforts to approach perfection are being made more difficult.
Well, it was something related to using AdGuard's (94.140.14.14) DNS and pool.ntp.org. I added Google's (8.8.8.8) to resolv.conf temporarily to get the clock synced. But I'm still not sure what the root cause was.
Unless someone is actually aiming to do a time coordinated task with others, you really need to know the time, or even the day-of-week to do stuff, because most human things in world aren’t actually occurring in small time sensitive windows.
I could reflash the OS, but I also "don't have the time". I'll just use my laptop until I do have the time someday to resurrect it.
You'll have to ignore DNSSEC time based validation while you start up, of course... Start with the last time you were properly synced: it's most likely less than a month behind then.
Then take a sampling of certificates offered from well known sites. If the certificates validate, other than time, no reasonable CA that you trust issues not-before dates in the future, so it's not before the most recent not-before date in a certificate. It's probably not much after the not-after dates either, well know sites tend to replace their certificates (but maybe you're getting MITMed with a compromised, old certificate).
Maybe ask for OSCP stapling, which should get you a more narrow range of times. I think OSCP responses are valid for a week? But if you get several from different https servers, chances are good you'll narrow the range.
In the case as suggested that the time was only off by about a day, IMHO, something is seriously wrong if it can't get to sync from there, but I guess I'm not really surprised either.
Years ago I've tried privilege escalation exploit to play with a phone and it involved rolling back date to unexpire signature, so I know there is exploit potential, but it... it just feels like RTC bootstrap problem should be something solvable.
Security protocols (at least the ones in common use) require certificates or keys that eventually expire, because of the risk of a permanent key being compromised. If they expire, the protocol needs time. QED.
While systemd-timesyncd contains a workaround for this, chrony does not seem to.
Plus the time to learn a few modprobe commands. Perhaps run a few commands echoing numbers to random sysfs files. Perhaps learn WTF a 'Device Tree Overlay' is - a concept that's completely absent on x86 systems. You see, this is an 'I2C' device, so it doesn't plug and play like USB and PCIe devices do. It is much simpler, and hence for some reason more difficult. Perhaps your device tree overlay needs to be complied? Or maybe not. Perhaps you'll edit your modules file and your kernel command line. Of course that won't involve update-grub, why would you think that?
Don't forget to periodically check on your settings, just in case an OS update or something has disabled one of those settings.
Oh and of course you won't be able to fully test this over SSH, if the network is up NTP will have set your clock whether the RTC is set up right or not. As we've learned from the article, if you run into clock problems you might not be able to connect remotely.
Then simply invent a time machine so you can do all of this in the past, before the pi SD card fails and it drops off the network.
/s
I learned about the devicetree on a RPi too, but I didn't complain about it in a blog. When I bought my first RPi I expected to encounter problems and learn a lot from solving them. In fact, I expected it to be much much worse - so much worse that I even bought the RTC module even before the RPi arrived in the mail. Why didn't he/she think of that? It's because he/she is lacking the basics, the kid project with gpio leds and i2c sensors.
(And you can do the same thing were you write the time to the RTC before you have the correct time)
We found they drift less than 28s a year in standalone mode. Also, were repeatable within +-18ms when using a NTP client.
Microchip RTCs are harder to setup, but also seem to work just as well (ignoring the goofy epoch).
Chip shortage caused people to do unspeakable things.... ;-)
Are you sure you connected it correctly?