[news] Emory University's SCCM server accidentally reformats all Windows PCs and servers

[Arthur]

This must be every system administrators worst nightmare! Luckily only 20% of all computers managed by SCCM at the university were affected.

 

Source: Emory University (via The Next Web / Reddit)

 

Have you ever reformatted a computer and then immediately realized you shouldn't have? Well, an "accident" at Emory University this week will make your mistake look like a brilliant, carefully considered decision.

 

Here’s the crux of what happened:

 

A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition/reformat set of tasks. As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

[ADMaster]

This is precisely why I have not moved to a ZTI deployment with SCCM and stick to my LTI with MDT.

[Sam_Brown]

If you are concerned about SCCM it's worth bearing in mind you can still do LTI with SCCM. It's what we use at work. Totally get why you wouldn't want to do ZTI though. Horror stories like this are why I refuse to implement it.

[synaesthesia]

Wow - as above, ZTI worries me a little. The servers/services only do what they're told, and the "telling" job is down to most commonly faulty component in all networks; human beings. Accidents happen, and no doubt those guys will be beating themselves up about it.

Prefer to just go to a machine and hit F12. Not quite as convenient if your based 3 miles from the furthest campus of course but I'm fully aware how likely I am to cock up!

[Norphy]

It must have been one heckuva balls up if the ConfigMgr rebuilt itself!

 

As the previous poster said, this was clearly due to human error rather than any deficiencies in ZT deployment. I'd bet he set the deployment against the All Systems collection without any kind of filtering on it. Madness!

[free780]

You do have to go through about 5 pages of dialog box with a required task sequence this should prevent a bunch of pcs getting imaged that shouldn't. That being said leaving those task sequences as avaliable gives you flexibility if you need a 32 bit image etc.

[Gatt]

I'm guessing it got deployed to the "All Systems" container

 

With our nice shiny new 2012 R2 deployment, I'm using RBA and Security scopes - All the built in collections are restricted to the SCCM admin and a few choice accounts.

Our desktop team can only see Workstation clients, and the server team can only see servers..

Edited May 17, 2014 by Gatt

[ADMaster]

Yes, I’m aware I can do lti with SCCM and I think I can do zti with MDT if I wanted to. The point is, this kind of story is why I stray away from zti.

I’ve made mistakes with SCCM already in the year I’ve had it. I deployed an app to all staff and made it required when it should have been available. Perhaps that is what they did only with an image.

 

Not to derail the thread too much but what I really want is to tell my MDT task sequence to install SCCM applications.

The pxe setup is nonstandard and trying to switch to sccm pxe will most likely break it.

I experimented with trying to get my WDS to boot into the SCCM environment with little success.

All that should be a thread to its self though.

[Gatt]

Yes, I’m aware I can do lti with SCCM and I think I can do zti with MDT if I wanted to. The point is, this kind of story is why I stray away from zti.

I’ve made mistakes with SCCM already in the year I’ve had it. I deployed an app to all staff and made it required when it should have been available. Perhaps that is what they did only with an image.

 

Not to derail the thread too much but what I really want is to tell my MDT task sequence to install SCCM applications.

The pxe setup is nonstandard and trying to switch to sccm pxe will most likely break it.

I experimented with trying to get my WDS to boot into the SCCM environment with little success.

All that should be a thread to its self though.

 

If you have SCCM 2012 R2 and integrate MDT into it, then you can use MDT task sequences in SCCM, and I believe one of them is a check to ensure you dont overwrite a Server OS with a Workstation OS..

[ADMaster]

I was testing out that integration yesterday, but I think I want it to go the other way around.

I read a site about converting a sccm bootable CD into a bootable wim for WDS so I can use my existing PXE settings.

But before I do that I wanted to make sure it would work, so I booted a VM with the ISO and had to configure a static IP, before it would find the task sequence.

I’ve not taken the time yet to search for an answer on that one.

Currently my PXE is WDS that boots the MDT images, all of my OS images and drivers are in MDT, and my applications are in SCCM.

It is a low priority project right now, I’d like to have it all intergraded, but if it doesn’t get done I can still image with MDT and deploy the application separately with SCCM.

[computer_expert]

At least they can say they have made progress migrating from XP (if they had any XP boxes left!)

 

As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.

 

I wonder if the task sequence completed?

[Gatt]

At least they can say they have made progress migrating from XP (if they had any XP boxes left!)

 

 

 

I wonder if the task sequence completed?

 

The task sequences would be fine until the servers (SCCM, DNS, DHCP & AD mainly) started.. at which point they would have all failed...

[free780]

Or what happened when it tried to update the db/logs?

[garethEds]

I know it's all quite serious but I did chuckle.

[free780]

It's funny the sccm server got wiped out. Presumably they have a different mp/dp or the TS would fail.

[cpjitservices]

Wow! - Jeez... can you imagine coming into work to find Windows 7 sitting on one of your servers.

[Gatt]

It's funny the sccm server got wiped out. Presumably they have a different mp/dp or the TS would fail.

 

The TS would have failed anyway as soon as a set of servers needed for the TS started the deployment

I suspect that up to that point then all the clients that received the TS would have at least wiped their drives and some may have started installing Windows 7 if the WIM file had already been downloaded. But once the SCCM server had been switched off, then any clients that had not received the deployment would be fine, and then it would vary to clients with a newly formatted HDD but not OS or at least a partially installed OS...