Findings Report on the July 8th Telstra Outage [pdf]

>This chassis was located in Melbourne and it was part of the mobile core timing architecture and was designed to provide stratum 2 and stratum 3 NTP services. When this server went back on line (at 2:50am), the GPS card in the chassis had reset during the change and began sending the incorrect date from 2006 into the network. The GPS card was actively used in that chassis and was missing a critical firmware update that was required to prevent this shift in time due to a known GPS rollover that was published in bulletins by the supplier. While the planned change of the NTP server chassis was the trigger point to begin the Outage, the steps leading to that point go back many years, as far back as 2020.

The core issue however was accumulation of technical debt (architectural changes in how timing systems were managed were not being tracked) and not recognising NTP as a sovereign system that needed to be properly managed.

> When Melbourne NTP was connected to the GPS card a ‘Network at Risk’ ticket should have been raised to find the real root cause of the frequent loss of the Sydney source and restore the NTP server back the intended design but this did not happen.

> At the time of the October 2025 incident, the personnel involved did not recognize the meaning of Melbourne NTP at stratum 5; that looping was already occurring in mobile core timing when an intended NTP stratum 3 server was able to become a client of one (or more) of the clients and that a circular loop was established when the stratum 2 source was lost. This anomaly was observed but not noted and no problem ticket was created for further investigation.

Two opportunities to raise tickets and nothing was raised.