Cloudstrike, Azure, Google, etc. When cloud computing goes wrong...

[Koldov]

This is how it happens isn't it...? All the dystopian sci-fi films about how either Skynet* becomes self-aware and takes over the world or some sort of (computer) virus or solar flare disables all our technology and throws us into the dark ages again.

 

It's the future (I know, I know), but I just can't shake the feeling we're pushing ourselves forward into some self-fulfilling prophecy by making more and more of our critical functions internet based and 'connected'... I do try to temper our general enthusiasm for cloud computing with the reminder that there is no ethereal 'cloud'... it's a physical thing, it's just basically someone else's computer. I mean sure, it's spread around, it's got failover/resilience/redundancy, but you are still going to be susceptible to the issues of bad OS/Anti-Virus updates (and other issues which take out ALL of the servers) and internet connectivity problems, etc.

 

Obviously, the more systems you depend on that are in the cloud and the more platforms that you store data on that are internet based, the more reliant you are on their availability to be able to function (never mind trust them to be secure)...

 

What happens when the fix isn't as easy as restart in 'safe mode' and delete a file...?

 

Hopefully I'll retire before I'm forced to give up my physical on-site servers (SIMS, File Shares, DCs, WSUS etc.) and local MS Office installs... I work in IT, so I'm no luddite... I'm mean I'm no Sarah Conner, but I guess with that attitude, I am probably the old-school dinosaur that the 'future' can't wait to get rid of...

 

*other AI's are available...

[Oaktech]

People ask me why I have very little tech in my house...

 

One of my colleagues couldn't turn the lights on in her house this morning because the app that controls them was down, and is still down, we assume due to a knock-on from crowdstrike.

[localzuk]

I think part of the problem we have is so many organisations love to put all their eggs in one basket.

 

Like that pension fund that lost all their data because Google deleted their account on Google Cloud. They had utilised all the resilience tools available within Google's system, but it was still just one system ultimately.

 

The IT industry needs to accept that if they really do want resilience, they need to start thinking about multiple suppliers for things. Spread it out.

[kevin_lane]

I think you are right on what your saying, I think sometimes we may get into the rut of “just put it into the cloud” like we have moved the issue of some sort.

 

Our ever growing relationship and reliance on technology is now more than ever more great, and we must not forget this, as today has shown the whole world (literally in some sense) has crashed.

 

And now people are panicking, I think sometime we need to do a reality check about how everything is integrated, because now we are at the point we are seeing smart IoT becoming a thing inside peoples houses, and our reliance is so great on being….connected

 

We need sometimes take step back and as my slt leader would say adjust the optics.

 

I’m sure we will all get through this with a bit of cloth and spit and mucking in and getting it sorted but lessons need to be learned and we should proceed with caution.

[mikes]

yeah I went to delete an old Shared Drive I made on google and it said "This operation is irreversible and all data will be lost" which spooked me as we have a lot of important stuff on shared Google drives and if someone accidentally deleted a shared drive and I wasn't able to recover it we'd be in big trouble. We already have backups, time to look into remote backups now for o365 and google. So is this cloud stuff really saving us money if we have to pay so many license fees it ends up costing more?

For now looks like I'll be spending Monday downloading zip files of all our shared drives manually...

 

Edit also reading about this Crowdstrike outage, it seems to me a lot of the times a lot of this "endpoint protection" (what we used to call antivirus I guess) is total garbage as it is, I'm sure I've heard about faulty antivirus updates deleting Windows files making machines unbootable before so this isn't the first time something like this has happened. You'd think Microsoft Windows would be more robust as it is

Edited July 19, 2024 by mikes

[jthompson]

Orgs using a diversity of systems to avoid having all of their eggs in one basket is a fine ambition, but ultimately there will still be some things higher up in the supply chain that represent a common point of failure. The trouble is that some of that stuff will be way outside of an organisation's ability to choose, or to even know about. Some kind of CDN or DNS or certification authority failure.

 

Am I right in thinking that flight control systems onboard aircraft are run with multiple systems in parallel, each written using a different language?

 

Extending that level of resiliency to an airport is obviously not feasible, but at least having management systems using different cloud computing providers, operating systems and AV, etc. seems like something achievable.

[psydii]

Our Business Continuity Plan requires diversity within our back-end platforms for precisely this sort of thing (the azure wobble). We have two telecoms providers, two sms providers, and can get to core student and staff data through three platforms. Files and email are all on a single provider, but with local caches of 'hot' data we can survive a temporary cloud outage. We could even bring email up with on-prem dialtone mailboxes if our cloud provider looked like it was going to be down for too long. Key cloud mailboxes and shared folders are backed up separately and could be migrated to the other provider of these services for education with "relative" ease, not that that's relevant in this particular scenario.

 

But I'm definitely reviewing our endpoint protection diversity following this, I need to make sure a bad update can't take down our imaging/software deployment platform *and* our end-user devices at the same time. Not sure whether the cloud-platform providers would be happy to confirm what their endpoint-equivelent protection platforms are.

Edited July 19, 2024 by psydii

[googlemad]

I don't think today is a 'cloud' issue, a security solution released a definition update and machines received it...could have theoretically happened with any AV/security vendor

 

We had a similar issue several years back where a McAfee drive encryption bug started to blue screen Windows 7 32 Bit machines and funnily enough the solution was broadly similar to this one (boot into recovery mode, run a command etc)...but to get into recovery mode involved the pain of manually having to decrypt the drive encryption first!

[MrEprise]

I don't think today is a 'cloud' issue, a security solution released a definition update and machines received it...could have theoretically happened with any AV/security vendor

 

Depends which way you look at it. It did affect some users in as much as various 'cloud' services were unavailable for those people who rely on said 'cloud' services.

[psydii]

Yes the cloudstrike element of the last 24hrs is not intrinsically a cloud problem, but its impact is due to broadly the same flaw as relying on a single cloud provider - a lack of diversity in the eco system. Though one might argue the minimal impact to most education establishments in the uk suggests perhaps we've got enough - but is that because edu (mostly) can't afford cloudstrike's products, and the 365 outage was basically overnight and mostly in the US?

 

That said, looking simply at the timing, and the (fairly) swift recovery/rollback, it wouldn't surprise me if it turns out the 'configuration change' in azure that brought US-Central down along with a significant chunk of 365 may have been the deployment of VMs with that contained the cloudstrike update.

[Oaktech]

You'd think Microsoft Windows would be more robust as it is

 

I think, despite apparent advances in baked-in endpoint protection, Microsoft still struggles with the concept that securing the fundamentals of their OS is actually their problem and not something that they can slopey-shoulder to their customers.

 

A 3rd party app being able to break the OS is case in point, but also the fact that there is MFA support in M365 and lots of evangelizing about it, but the MFA support within the OS itself can be best described as patchy!

[dmj]

I think, despite apparent advances in baked-in endpoint protection, Microsoft still struggles with the concept that securing the fundamentals of their OS is actually their problem and not something that they can slopey-shoulder to their customers.

There's a pertinent article about how internal powers at microsoft actively block security initiatives, in this case leading to US govt compromise in the 'solarwinds' incident. https://www.propublica.org/article/microsoft-solarwinds-golden-saml-data-breach-russian-hackers

[mikeprice]

People ask me why I have very little tech in my house...

 

One of my colleagues couldn't turn the lights on in her house this morning because the app that controls them was down, and is still down, we assume due to a knock-on from crowdstrike.

 

That is hilarious

 

There must be people who can;t make a cuppa because the kettle app is down!!!

 

 

but also scary!

 

The lights in the house is minor - but scale it up and it becomes patients dying because apps can;t switch machines on in a hospital

[bknaggs]

Does anyone else suddenly feel out of their depth because they hadn’t even heard of Crowdstrike before Friday?

 

Imposter syndrome isn’t something I normally have but it’s appeared these last few days.

[timbo343]

Does anyone else suddenly feel out of their depth because they hadn’t even heard of Crowdstrike before Friday?

 

Imposter syndrome isn’t something I normally have but it’s appeared these last few days.

*Hand up* I've seen CrowdStrike advertised on the side of racing cars but even i had to Google CrowdStrike on Friday to see what they did.

 

It doean't help when the media get hold of IT issues and everyone becomes an IT expert.

Edited July 21, 2024 by timbo343

[nicholab]

I think one of the issues is in the cloud we can build redundant systems on premises we were never allowed to budget wise. This is why the cloud is attractive to education IT teams.

 

I have the issue that down the road is data centre the company I work for never used it now we are migrating to Azure.

Edited July 21, 2024 by nicholab

[gaz350b]

The problem you have with something like crowdstrike or similar EDRs is that you are essentially collecting data from end point threats to get an organisation wide picture of what is going on.

 

You simply can’t use multiple platforms to that. It’s not just about malware it’s also about if Joe in finance is suddenly doing something that isn’t deemed normal behaviour. Trying to login to 100’s of servers from his device etc and that applying a response (most likely automated).

Edited July 21, 2024 by gaz350b

[thimon]

Does anyone else suddenly feel out of their depth because they hadn’t even heard of Crowdstrike before Friday?

 

Be thankful you hadn’t. All the schools in our Trust use it. Good times…

[paulkerton]

The problem isn't where your servers are stored. The idea that a local server wouldn't be affected by something like this is asinine as that is literally what happened in thousands of businesses around the world. It wouldn't matter where your data was stored. Having a dusty old machine in a cupboard in a primary school classroom isn't going to save anyone from what happens when a bad update is issued.

 

The immediate issue is Crowdstrike's lax policies regarding updates.

 

The big massive wider issue that everyone wants to seemingly actively avoid is having an operating system where Ring 0 operations are allowed to be ran by a third-party piece of software using a hardware driver to do so which can bring the entire OS to it's knees and render it useless.

 

Which of course can't really be fixed, because it will then lead to cybersecurity vendors suing Microsoft on anti-trust reasons because Microsoft have basically managed to create a whole industry around their OS not being secure-by-design.

[FN-GM]

Does anyone else suddenly feel out of their depth because they hadn’t even heard of Crowdstrike before Friday?

 

Imposter syndrome isn’t something I normally have but it’s appeared these last few days.

 

It’s probably not on the radar due to the cost. It’s very expensive. But works really well! Stopped our pen testers, they couldn’t get in due to it. Also seen it stop an attacker on a test network. I managed to trigger it and isolate a server when I was testing the security I implemented. It’s not simply an anti virus as many think.

Edited July 22, 2024 by FN-GM

[psydii]

Microsoft release a new recovery tool to assist in recovering machines impacted by the CloudStrike update.

 

https://techcommunity.microsoft.com/t5/intune-customer-success/new-recovery-tool-to-help-with-crowdstrike-issue-impacting/ba-p/4196959

 

This new recovery tool can be run from inside WinPE from existing PXE Boot infrastructure such as Configuration Manager, used to create an emergency PXE Boot server for the purpose, or booted from USB that it helps you build.

 

Even if you don't have cloudstrike, I think this tool might be worth a look - just to see how it's put together, some of the techniques it employs might be handy for solving other jobs around the place!

 

Of course you still need to be be able to get to the bitlocker keys - no keys, no boot. (Though there have been rumours in some configurations safe mode is reachable without keys, allowing for recovery with the local administrator password, but without the bitlocker keys.)