Telstra outage: Damning report finds staff were on ‘mandatory standdown’ at time of mass system failure

An investigation into a catastrophic national outage in July that left millions of Telstra customers without mobile services has revealed two engineers primarily involved in the ultimately bungled software upgrade were on “mandatory standdown” at the time.

The damning external report released on Wednesday also concluded the telecommunications giant has “too few engineering staff and insufficient expertise” to support its NTP platform — the network time protocol technology that’s used to synchronise time across the network.

The findings by Technology Audit Partners shed more light on the disastrous 12-hour outage, which also resulted in more than 600 calls failing to connect with Triple Zero services.

The report found the outage was caused by an incorrect date propagated through parts of the telco’s mobile network following a planned system update.

However, TAP concluded the outage was “primarily a result of (Telstra) not treating network timing as a critical capability within the network … requiring the highest levels of oversight and protection”.

It also took aim at Telstra’s lack of critical expertise.

“On the night of the outage the two primary NTP engineers who were involved in the planned change on the chassis were on mandatory standdown by the time the impact of that change became known,” it said.

“Overall, there is not sufficient knowledgeable staff to do all the necessary work and peer review for NTP to fulfill all product ownership and support responsibilities.”

Telstra services went black about 2am on July 7 with no estimate given of when they would be restored. The outage also affected the Aldi Mobile, Belong and Boost brands.

TAP also uncovered inadequacies in identifying technical expertise for NTP systems, with Telstra being told it needed stronger “competency and more curiosity to investigate when something looked wrong”.

It also found Telstra had shortcomings in “process discipline including change management, configuration control, incident management, documentation and follow through”.

The incident response was lacking with gaps in ownership and visibility, while operational support of the network timing system delayed the identification of the problem in the early stages of the outage.

Telstra said it had made changes in the wake of the outage, including reviewing critical non-timing-related functions across its network and reviewing its vendor notification management processes to improve how future alerts play out.

It has also assessed its “alarm instructions and service assurance arrangements to ensure our technical teams can action all issues without delay”.

CEO Vicki Brady on Wednesday revealed a further eight calls to emergency services were missed that weren’t initially subject to a welfare check as per protocol.

“It shouldn’t have happened, we should have picked those eight calls up,” Ms Brady said.

“The required checks have since been completed, and Telstra is not aware of any adverse outcomes for those callers,” she said. Ms Brady accepted the network did not adequately prioritise the network timing system.

“As a result of prioritising it not at that highest level, that had some flow-on impacts, which included the TAP findings identifying in this particular area in the teams looking after this network timing system.

“We did not have sufficient resourcing and expertise. Just to be clear, we have reallocated already resourced staff into that team with the expertise required, and also drawing on external experts to make sure we have that capability at the right level.”

Almost $1 million across more than 30,000 customers has so far been distributed in credit after the outage also affected payment services.