Dive Brief
- A Microsoft 365 outage that cut off Exchange Online mailboxes worldwide for nearly a day starting Nov. 12 was due to a defect in a Windows fix Microsoft began rolling out to networking gateway servers in July, Microsoft EVP of Copilot, Agents and Platform Charles Lamanna wrote in a November 16 post on X. The change ran worldwide for more than three months without problems, according to Lamanna.
- About 15 hours before the outage, Teams failed over to the gateway that also serves Exchange Online. A separate problem the day before had already taken nearly 14% of that gateway's capacity out of service. At the worst point, more than 80% of the gateway servers stopped responding, and because the fix was several months old, Microsoft couldn't simply roll it back, according to Lamanna.
- This is the fourth major Microsoft 365 outage this year tied directly to Microsoft's own changes and maintenance. A final report on this latest incident is due November 20.
Dive Insight
The risk for partners isn't in anything they can see, test or delay. The fix that broke Microsoft 365 passed the company's own safe deployment process, ran worldwide for months and failed when an ordinary traffic shift landed on a gateway that was already short on servers. MSPs and CSPs took customer calls but had no warning.
The details in the preliminary report, linked here and also summarized by online forums, point to gaps that load testing didn't catch. The defect caused lock contention inside an internal process that made servers unavailable, and it worsened on newer machines with more cores. Lamanna wrote that Microsoft's "load testing tended to focus on our weaker hardware SKUs with fewer cores." Safeguards meant to keep a failure inside one region also broke because of a separate issue, causing the problem to spread widely within a couple of hours. Teams, SharePoint and OneDrive could fail away from the bad gateway, but Exchange Online apparently could not.
The pattern is what partners should raise with customers. In January, Microsoft blamed a North American outage on an “elevated service load” during maintenance. In July, a bug in automated network maintenance pulled routes from more devices than intended. The August outage, which took more than two days to fully resolve, came from a code update to Microsoft's certificate management system. CIO Dive and Computer Weekly both reported last week that analysts see the November outage as a warning about provider concentration.
Now we look ahead to November 20, when Microsoft's final report is due. The preliminary version lists added capacity, monitoring for traffic redirection and guardrails on how traffic is distributed. What we don't know is whether Microsoft will change how it tests infrastructure fixes against production hardware.