Microsoft 365 Suite Incident

[tangotech]

User Impact: Users may be unable to access multiple Microsoft 365 services.More info: We've received reports that the following services are impacted:-Microsoft Teams-Exchange Online-Outlook-SharePoint Online-OneDrive for Business-Microsoft GraphCurrent status: We've identified a potential networking issue and are reviewing telemetry to determine the next troubleshooting steps.Scope of impact: Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services.Next update by: Wednesday, January 25, 2023, 9:00 AM (9:00 AM UTC)

[FN-GM]

It’s a global issue. Also impacting Australia.

[mitchell1981]

Can anyone else confirm if it is affecting hosted / connected Sims? Do they use Azure servers? Ours is working very intermittently, and their status page appears to have gone completely! Causing chaos with cover.

[HPlum78]

Yeah the issue with having the status page behind a login is that when everything is not working you cannot see the notifications!

[tangotech]

No reports but yes we are also having issues with our Azure servers....

[HPlum78]

And I do believe that hosted sims is in Azure.

[HereIGoAgain2601]

Our hosted Sims has been impacted

[Oaktech]

We're seeing random impact to our users of OneDrive and Outlook, nothing more so far.

[Olliedawg]

No issues to report at my workplace as of yet.. Although I am unable to access the Volume Licensing Center which is a PITA

[Andrew_C]

"Move everything to the cloud" they said. "It is safer that way" they said.... FFS.

[Andycat]

Was a nightmare 6-7.45am, but seems ok now. Network change as usual they say....

[HPlum78]

@Andrew_C if no one can get to it, is that not safer? MS has just got us all cyber essentials certified...

[psydii]

Opened up my inbox this morning to a blizzard of SAN error alarms, and then it went silent.

 

SAN turns out was 'fine' (degraded but fine), it was just that 365 went down in the middle of the SAN telling me its problems making it look like the SAN had completely died.

 

Teams and Outlook were back up for us by about 0820UTC, most people have their files cached by the onedrive client so we are shielded from the current lingering problems on the sharepoint side of the farm.

 

SIMS sits on our SAN so I thought I was going to have a very bad day.

[ITGuyNW]

They're blaming it on a network configuration change and have now rolled it back. Must have been some change.

[atcoates]

Microsoft says it has "rolled back a network change" it thinks may be linked to tens of thousands of worldwide users being unable to access its services, including Teams and Outlook.

 

Downdetector, which tracks website outages, showed more than 5,000 people in the UK had reported the email service Outlook was inaccessible.

 

Other services including Teams and Xbox Live were also reported as not working.

 

Microsoft said it was "monitoring the service" as the change takes effect.

[MartinT]

"We’ve identified that a wide-area

networking (WAN) routing change caused impact to the service. We’ve rolled back

the change and monitoring the service as it recovers. Some of the customers who

had previously reported impact are also reporting recovery."

 

Sounds like someone is in trouble.

[Olliedawg]

Would not want to be that guy/girl right now who authorized the change

[psydii]

Generally that's not how things work in the big cloud providers. Mistakes happen. Systems and processes exist to reduce the chance of a mistake getting into production, but they are inevitable. What is learned is what needs to be done to prevent that same human/systems failure re-occurring. If a human pushed the wrong config to the wrong system, then a system needs to be put in place to prevent either that happening again, or happening at such scale so quickly again.

 

Individuals come and go, so there is no point in singling someone out because blaming a specific person prevents the *organization* from learning. Because what ever happens, the next outage will be be triggered by the actions of a different human.

Edited January 25, 2023 by psydii

[HPlum78]

Yep you made a mistake if I get rid of you then that's our loss if I keep you this will not happen again. Also let's be fair here there is no one person involved in a reconfiguration of how the underlying routing works in the MS cloudy environment. This has passed a few people's eyes and probably some AI before they took down their entire operation!

[JATSO]

"Move everything to the cloud" they said. "It is safer that way" they said.... FFS.

 

That's the best comment yet.

[LeMarchand]

Anyone having problems with Link to Windows today? My phone has just lost all my linked devices and won't connect to the one in front of me. Wondering if it's connected to these problems.

[MartinT]

Anyone having problems with Link to Windows today? My phone has just lost all my linked devices and won't connect to the one in front of me.

 

Co-incidentally, my link (Android to Windows) broke a while ago and I re-made it last night.

[andy_b]

Hive (Heating) seems broken today too

[LeMarchand]

Co-incidentally, my link (Android to Windows) broke a while ago and I re-made it last night.

 

 

Yeah, just had to do the same. Just my house and other two sites to go, I guess.

 

Hive (Heating) seems broken today too

Just checked. Mine seemed OK, then threw a wobbly and settled down. Doesn't look like the adjustment I made to the "work day" temp this morning as everyone is working "out" today took, though. The cat will have been happy, at least. Looking at the usage chart, it's not him wanting the door open to go out that's causing the high usage. Weeps.

[FN-GM]

"Move everything to the cloud" they said. "It is safer that way" they said.... FFS.

 

Outages happen on prem as well...