London

June 28–29, 2027

New York

September 15–16, 2026

Berlin

November 9–10, 2026

Microsoft’s outage started with a fix

The bug testing never caught.
November 17, 2026

Estimated reading time: 4 minutes

Key takeaways:

  • Make your test environment match production. Microsoft tested on less powerful hardware and a defect that surfaced on servers with more cores went unnoticed for four months.
  • Check what your health checks remove because automated failover pushed traffic onto working servers until more than 80% of Microsoft’s gateway was unresponsive.
  • Rehearse the recovery too, as restarting servers with traffic waiting is rarely tested, so practice a 24-hour Microsoft 365 outage, runbook included.

A huge outage late last week lasted far longer than the company wanted, but what lessons can software engineers take from the issue?

Twenty hours is a long time in anyone’s book. It’s an extraordinarily long time for a service as ubiquitous as Microsoft’s suite of products, including email and countless other tools, to remain offline.

Yet that’s what the tech giant faced late last week as an unexpected outage caused chaos as millions of people were taken offline, alongside allied services. “Large cloud providers run enormously complex networks, and a single bad change can still ripple much further than anyone planned for,” says Corey Nachreiner, chief security officer at WatchGuard Technologies. 

Just as consequential as Microsoft’s systems going offline was the time taken to bring them back online. The company appeared to struggle to get a handle on its own systems, and it took the better part of a day for normal service to resume.

A postmortem has pinpointed the issue as a software defect introduced in July, which only became apparent when three separate circumstances combined to trigger a global failure. However, the incident highlights a broader issue. 

“The outage also reinforces that recovery does not always happen as quickly as planned,” explains Dennis Martin, crisis management and business resilience specialist at Axians UK. “Even the largest technology providers can encounter unexpected challenges when restoring complex services at scale,” something Microsoft found out to its embarrassment. 

Long outages, longer-term issues

“Business continuity plans should account for extended disruption, including practical workarounds, alternative communication channels, and a clear understanding of what minimum viable operations look like,” advises Martin.

The reason why is still being hashed out. Microsoft has released a statement, through Charles Lamanna, the EVP of Copilots, agents, and platform at Microsoft, who said the problem began with a fix to the networking gateway used by Microsoft 365. The change had been in place for months without issue, but under certain loads it caused competing processes to lock up, exhausting the servers’ CPU and memory.

That flaw wasn’t discovered in testing. Microsoft had used less powerful hardware in its labs, whereas the bug was more likely to appear on newer machines with more processor cores. It’s a painful lesson in assuming a test environment reflects production.

“What is striking here is that there was nothing to roll back,” says Jules Duvivier, director of security engineering at Upsun. “According to Microsoft’s preliminary report, the bug shipped in July and ran on its gateway servers for about four months, until a traffic failover and some missing capacity set it off. So they had to find the cause and write a fix in the middle of the outage.”

What caused the Microsoft outage?

The issue, according to Microsoft, was compounded by two other problems. Around 15 hours before the disruption kicked off, Teams traffic was also moved onto the gateway most commonly used by Exchange Online, which upped the load on that gateway. At the same time, around 14% of the gateway’s server capacity was out of service due to an unrelated issue. 

Despite that, the issue kept going. Microsoft’s automated systems responded to servers becoming unresponsive by pushing traffic to those still working, which promptly overloaded them too.

At the worst point, more than 80% of servers in the gateway were unresponsive. “The problem is that automated health checks don’t always distinguish between a server that’s completely unavailable and one that’s simply responding slowly,” says Panayot Kalinov, senior software developer at Casinoreviews.net. “Removing too many servers can make an already difficult situation much worse.” 

How to prevent the issue recurring

There are ways around this – ones that Microsoft has learned too late, but which may prove useful for other teams in the years ahead.

“To prevent this, engineering teams should test configuration changes on a small number of servers before rolling them out more,” says Kalinov. He adds: “I’d encourage engineering leaders to check whether their health checks could accidentally remove functioning servers during periods of high demand. It’s also worth testing how their systems behave when capacity suddenly drops.”

Even then, it’s not foolproof. Microsoft rolled out its July update gradually, yet the problem still went unnoticed.

Join LeadDev events in 2027

There’s also the issue of recovery. Microsoft says its servers were overwhelmed by waiting requests whenever they were brought back online, which meant engineers had to restore capacity in batches. “Restarting most of a fleet with all the traffic waiting to come back is a different problem, and one that is rarely tested,” says Duvivier.

Nachreiner advises organizations to rehearse a 24-hour Microsoft 365 outage, including testing alternative communication tools and ensuring critical information is accessible elsewhere. “Most teams discover during the exercise that their runbook lives in SharePoint,” he says.