Microsoft has identified the root causes of the global outage that crippled Outlook, Exchange and a host of other vital office software last week, according to an initial report issued on Monday.
The report identified three factors that led to the 24-hour blackout of its services: a months-old bug, malfunctioning internal connections, and server capacity. Each of these factors may have easily been resolved on their own, but, when taken together, they led to a major outage that was harder to mitigate, Microsoft executives said.
Sequence of events
The disruption began Thursday with US users reporting interrupted access to their Outlook email accounts. As it continued into the next day, millions of people worldwide said they could not access their email in any capacity as well as other Office 365 services. Jails could not release prisoners, hospitals turned to paper records, schools lost contact with parents, and police departments could not receive crime reports. Microsoft said it had resolved the issue and restored service by Saturday, also issuing an apology. The company did not attribute the blackout to a cybersecurity breach.
In a post on X on Monday, Charles Lamanna, the executive vice-president of Copilot, Agents, and Platform at Microsoft, said there was a latent bug or defect from July that caused Microsoft’s servers to “become overwhelmed and unresponsive”.
Lamanna wrote that the company typically has two available networking gateways, or entry and exit points that connect separate systems and allow data to pass between them. However, one of them was not working for Microsoft Teams which overwhelmed the primary gateway. Subsequently, over 14% of the company’s server capacity was also not working due to an unrelated issue from the previous day.
“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July,” Lamanna wrote. “This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.”
The company also had trouble mitigating the bug because it originated months before, Lamanna said, so it would not be easy as rolling back a recent change to undo the defect.
“Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again,” he wrote.
Sources: rss_autopost