Man. Time based bugs are some of the worst. I spent weeks trying to figure out what was wrong with some of my scripts running under WSL. Apparently wsl Linux kernel had a bug that could cause time to drift by minutes.
Every now and then, certain apps on the a system would crash around the same time. We'd scrape through the logs and usually see our app cratered, some databases, ntpd, sshd naturally. Logging of timestamps was iffy, obviously.
The ntpd was the obvious suspect because, despite keeping a nice low offset to its peers like 1ms or two, for months at a time, out of the blue it would confess something like "time offset is too large, I can't fix that so I'll exit!". After chasing Sun ntpd bug reports[1] for a while, we ruled it out when we saw a pattern in the undamaged logs that looked like
09:59:58.000 ...
09:59:59.123 ...
09:09:01.345 ...
09:09:02.123 ...
After some Sun support and some sunbugs searching [1 again] we found the T4 in that Solaris rev had a hardware RTC with separate registers for H, M, S etc and a write mutex protecting them, but no read mutex. It was possible to read the RTC while it was being updated, which happened when it was syncing the OS clock to the RTC, or something like that. Fixed in a later release.
1: RIP sunbugs database. It was such a mature relationship where Sun would let everyone see what they were working on and customers could participate or at least know about known issues. I would love to find an archive. Of course Oracle shut that off immediately so you had to open a ticket and ask.
Someone on a team I was on spent probably weeks working on a way to process an inbound sensor data stream (which was generated by another bit of software also maintained by us). The data was akin to an odometer, increasing over time based on usage. The trouble was there appeared to be two parallel streams of the same name, which were always offset from each other and while the offset varied they were always roughly close. The algorithm this person came up with eventually sorted them into "thing_1" and "thing_2" streams, at which point it got turned over to me to display. It actually worked pretty well, for what it's worth.
I started asking how am I supposed to show this to a user to make use of, what does this even represent?... but never got an adequate answer. So I started looking at the whole chain, and what I found was the piece of software generating the data had a small bug: it used "hh" instead of "HH" in the timestamp, but also no am/pm. The timestamps were supposed to be 24h and looked like it, but 9:02AM and 9:02PM both came in as "09:02". To confirm, we checked the database and, sure enough, every bit of data was between 1:00am and 12:59pm. In the end we fixed the timestamp bug and threw out the processing code.
It's simultaneously a hilarious date bug, a face-palming colossal waste of time, and a lesson on how not to run technical teams (the manager had no technical skill, and the "team" of ~7 was very siloed and each operated more like teams of 1-2).
Could you please share what did you read on this topic?
Thank you!
* - https://www.federalregister.gov/documents/2020/03/23/2020-06...
* - https://barrgroup.com/sites/default/files/case-study-patriot...
Clock is fucked, so TLS certs don't verify due to validity times, so DNS is broken, so NTP cant look up domains, so the clock can't be set...
On the one hand there's not very much else you can do since you're pointing at bits of thin air and saying "there's the trust chain" in the first place, but on the other hand plaintext DNS is... not much better?
Of course that's when the existential "why even DoH in the first place" starts (with side servings of "this feels so wrong putting it on the security report")...
(...Why do I suddenly feel like disabling certificate verification is going to catch on in a big way in embedded ntpds, almost like a standard best practice... aaaaaaaaa)