Microsoft Identifies Software Flaw, Traffic Surge and Reduced Server Capacity as Causes of Global Outage

0
4

NEW DELHI–Microsoft has identified a previously undetected software defect, unusually heavy network traffic and reduced server capacity as the three major factors behind a global Microsoft 365 outage that disrupted services earlier this month.

The company disclosed its preliminary findings Monday, November 16, following an investigation into the November 12 disruption, which affected Microsoft Teams, OneDrive, SharePoint and Office applications.

According to Microsoft, the outage originated in a networking system responsible for directing traffic to several Microsoft 365 services. A hidden software flaw caused servers to become overwhelmed and stop responding when certain conditions occurred simultaneously.

A Microsoft Executive Vice President responsible for Copilot, Agents and Platform apologized to customers in a post on X and outlined the company’s initial findings.

The investigation determined that the software defect had been introduced through a code update deployed in July. Despite operating across Microsoft’s global infrastructure for more than three months, the flaw had not previously caused significant problems.

Microsoft said the issue escaped detection because its testing environments used different hardware configurations and traffic patterns than those encountered in production systems.

The conditions that triggered the outage began developing approximately 15 hours before services were disrupted, when Microsoft redirected Teams traffic from a backup system to its primary infrastructure.

That change substantially increased the volume of requests handled by the affected networking system.

At the same time, approximately 14% of the available server capacity was already offline because of an unrelated problem that occurred the previous day.

Microsoft said neither the increased traffic nor the capacity reduction would have independently caused a major outage. However, the combination exposed the underlying software defect.

As traffic increased during peak business hours in the United States, requests began accumulating faster than servers could process them, eventually causing widespread failures.

Microsoft’s automated traffic management systems initially attempted to maintain service availability by redirecting requests away from unresponsive servers to functioning ones.

However, the additional traffic placed greater pressure on the remaining infrastructure, triggering similar failures across other servers.

The resulting chain reaction spread rapidly through the system.

At the height of the incident, more than 80% of the servers in the affected infrastructure had stopped responding, according to Microsoft.

The company also discovered that a separate technical problem prevented internal safeguards designed to isolate regional failures from functioning as intended.

As a result, the disruption spread beyond its initial location and affected Microsoft 365 services globally.

Restoring operations proved particularly challenging because servers that restarted were immediately overwhelmed by requests that had accumulated during the outage.

This caused some servers to become unresponsive again shortly after returning online, complicating recovery efforts.

Microsoft acknowledged that its existing automated recovery mechanisms were not designed to handle a failure of this magnitude.

Engineers eventually traced the disruption to the faulty software code and developed an update to address the problem.

The corrective update allowed Microsoft to stabilize its infrastructure and restore normal service operations.

The November 12 incident disrupted access to several widely used workplace productivity and collaboration tools, highlighting the challenges of maintaining reliability across large-scale cloud computing platforms.

Microsoft said it is continuing to investigate the incident and plans to publish a comprehensive review later this week.

The upcoming report is expected to provide additional technical details, explain the company’s recovery efforts and outline corrective measures intended to reduce the likelihood of similar outages in the future. (Source: IANS)