107 karma · joined June 29, 2015
An advantage of constantly loading at least one core of the CPU might be preventing the deeper power states from kicking in, which should make the RX timestamping latency more stable and improve stability of synchronization of NTP clients.
ntpsec as a project seems to be doing ok. They are releasing new versions, fix reported issues, accept patches, and develop the code publicly. While ntp still has a huge list of acknowledged but unfixed CVEs.
The author has other posts in the series where he tried to measure the accuracy relative to the PHC (not system clock) using PPS: https://scottstuff.net/posts/2025/06/02/measuring-ntp-accura...
Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets.
> So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this
That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).
The switches could also implement a proper HW-timestamping NTP server and client to provide an equivalent to a PTP boundary clock.
PTP was based on a broadcast/multicast model to reduce the message rate in order to simplify and reduce the cost of HW support. But that is no longer a concern with modern HW that can timestamp packets at very high rates, so the simpler unicast protocols like NTP and client-server PTP (CSPTP) currently developed by IEEE might be preferable to classic PTP for better security and other advantages.
- wifi is most reliable, secure, easiest to debug, but usable only for mains-powered devices due to higher energy consumption
- bluetooth LE has lowest energy consumption, best for unreliable broadcasting of data (e.g. temperature sensors), but has shorter range
- zigbee is best for battery-powered devices where reliable communication is needed and is initiated by the device (e.g. switch, window/door sensor)
- zwave is best for battery-powered devices which need to quickly receive data (e.g. door lock or directly controller radiator valves), but security seems problematic according to some reports
A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.
It seems they didn't replace the PTP offset calculation assuming a constant delay (broadcast model). That doesn't work well when the distribution of the delay is not symmetric, e.g. errors in hardware timestamping on the NIC are sensitive to network load. They would need to measure the actual error of the clock to see that (the graphs in the article seem to show only the offset measured by SPTP itself, a common issue when improvements in time synchronization are demonstrated).
I think a better solution taking advantage of existing PTP support in hardware is to encapsulate NTP messages in PTP packets. NICs and switches/routers see PTP packets, so they provide highly accurate timestamps and corrections, but the measurements and their processing can be full-featured NTP, keeping all its advantages like resiliency and security. There is an IETF draft specifying that:
https://datatracker.ietf.org/doc/draft-ietf-ntp-over-ptp/
An experimental support for NTP-over-PTP is included in the latest chrony release. In my tests with switches that work as one-step transparent clocks the accuracy is same as with PTP (linuxptp).
Rust code needs proper fuzzing too. It takes a lot of effort to ensure everything is covered and stays covered as the code is developed. Crashing libraries or applications can be a denial of service. Sure, it's lower impact than an RCE due to a buffer overflow, but it is still a security issue.
Measuring that error is difficult. A possibility is to run a program periodically making random reads from memory (avoiding the CPU cache) to generate a PPS signal on the memory bus, which can be observed with a scope. There is a lot of noise due to other memory activity, RAM refresh, etc. From the configured memory speed and tRCD+tCL timings the uncertainty of the error can be reduced.
This might improve with new hardware. There is a feature called Precision Time Measurement (PTM), which is a hardware implementation of an NTP-like protocol in PCIe, but so far I have seen this working only on some onboard NICs.
Both NTP and PTP don't care (as protocols) where the timestamps are coming from. That's an implementation detail.
> NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions
That was maybe 20-30 years ago, but not today. The wikipedia article needs an update. If you don't hit a routing asymmetry, in my experience it's usually milliseconds over Internet and tens of microseconds in local network if using SW timestamping. Please note that NTP clients by default use long polling intervals to avoid excessive load on public servers on Internet, so they need to be specifically configured for better performance in local networks.
You can find some measurements with HW timestamping here: https://chrony.tuxfamily.org/examples.html
Note that this is for the system clock, which has to be synchronized over PCIe to the hardware clock of the NIC. That adds hundreds of nanoseconds of uncertainty. It doesn't matter if the hardware clock is synchronized by PTP or NTP.
If you care only about the hardware clocks, it's easy to show how accurate is the synchronization by comparing their PPS signals on a scope. NTP between two directly connected NICs, or a with a hub, can get to single-digit nanosecond accuracy. I have seen that in my testing. It's just timestamps, it doesn't matter how they are exchanged.
PTP has good support in higher-end switches and routers, but it's difficult to secure and make resilient to failures. It was designed for automation and control networks in factories etc. NTP is a better fit for computer networks, but there doesn't seem to be any switches or routers with HW NTP support. If you really need the best accuracy with NTP, you can find old 100Mb/s hubs on ebay and create a separate network.
If you still have such a machine (preferably without any valuable data), try setting the date right before the overflow with this command
date -s @$[2**31 - 10]
and see if you can recover the system without reboot.
I have seen different daemons stopped responding and just consuming CPU, NTP clients flooding public servers, and other interesting things. I suspect many of these systems will still be running in Y2038 and that day the Internet might break.
- 8 were found within the project itself (mostly by me)
- none are memory-safety issues in the NTP-specific code
- the last memory-safety issue is from 2015 and it was in the custom management protocol (exploitable only by authenticated users), which was since then greatly simplified and made stateless
The project now has an excellent fuzzing coverage, it was audited, and I'm quite confident there are no remotely reachable memory-safety issues. I'll buy you a drink if you find one :).
NTP as a network protocol is extremely simple. There is no complex data, almost everything has a constant length. A minimal server+client implementation can be written in a few hundred lines of code. I wrote one in Rust, but the reason was server performance, not security.
Most of the complexity related to NTP is on the client side, in the processing of measurements provided by NTP. I don't think the language matters much here. However, if major operating systems will start switching to services written in Rust, I don't see a reason why chrony couldn't be rewritten in Rust, first the small parts related to networking and later everything.
But most operating systems already have their own mechanism to update their timezone database. This is not a problem. Shifting timezones by 15 minutes every few thousand years would work with current software.
All we need to do is to agree that UTC will have no more leap seconds.
Those don't care about the protocol as they can provide hardware timestamps for all received packets.
Other popular NICs like the Intel X540 or XXV/XL710 are limited to timestamping of PTP event messages in order to limit the rate of timestamps which needs to be handled by the driver. For those chrony supports an NTP-over-PTP protocol which forces the hardware to trigger the timestamping by wrapping NTP messages in PTP.
Sub-microsecond accuracy is certainly possible. Here is an example with 3 network switches: https://chrony.tuxfamily.org/img/client-hwts-3switch-f323.pn...
The accuracy is limited by asymmetries in network switches.
In any case, whatever algorithms Clockwork is using with their protocol, I'm sure they could be used with NTP too. If additional information needs to be exchanged between the hosts, extension fields can be specified for that.
Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.
There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.
The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.
Some switches support NTP as a server and client (equivalent to the PTP boundary clock), but there don't seem to be any using hardware timestamping. It's just the classic ntpd using software timestamps, good to few tens of microseconds at best.
And yes, NTP could definitely perform as well as PTP if the switches had a proper support. In my tests with directly connected NICs the synchronization is stable to few nanoseconds, same as with PTP. At the protocol level, they use the same timestamps.
A highly accurate stratum-1 NTP server can be build with a common computer NIC. No need to mess with FPGAs (unless that's your thing). The Intel I210 is about $50. It has a PPS input and output. With some calibration, the timestamping can be accurate to few tens of nanoseconds.
NTP can work very well between directly connected NICs. But without hardware support in the switches/routers, that accuracy degrades quickly in the network. A single switch can easily add hundreds of nanoseconds worth of jitter and tens of nanoseconds worth of asymmetry.
In PTP the problem is in the synchronization protocol itself. A master in the unicast mode is basically a programmable packet generator. It sends sync/announce messages at a rate and duration specified by its slaves, and the address can be spoofed.