Slow boot times, suspect MS Endpoint Protection

[disk]

For the past few weeks we have been getting a trickle of reports regarding slow system boot times. We are running mainly Windows 8.1, though there have been reports of this from some of those who are still running Windows 7. We have just started our migration to Windows 10 and none of our pilot users nor any of us here in IT running Windows 10 have been affected.

 

We are running System Center Configuration Manager Current Branch 1602 in our organization. We just began using SCCM for imaging a year ago, but we were using SCCM for just Endpoint Protection for the past 5 years. Some of the systems reporting slow boot times were imaged with Dell Kace with an Endpoint Protection client post install task. Others have been imaged within the past year using SCCM, again, with a post install task to install the Endpoint Protection client. At first we thought it might be a simple matter of determining Endpoint Protection client and/or definition version. This is where things get tricky. On one system with the latest definitions, the client was running 4.9.218. The version that we were currently installing for newly imaged PCs was 4.9.219. Upgrading that system allowed the PC to boot quickly for a number of restarts. Then the slow boot came back. On another system, it was running SCEP v4.10. Downgrading that one to 4.9.219 fixed the slow boot for several reboots and all seemed well. Left the PC on at the end of shift. During the night, the PC shut down via a scheduled task. Turned on the next morning and once again had the long boot issue.

 

We have looked through Windows Event Logs, SCCM logs, used Windows Performance Recorder/Analyser, and so far, nothing has been able to explain this.

 

One last thing of note.

 

On one of the PCs that was experiencing the slow boot, I installed the latest version of Faronics Deep Freeze. That system has been rock solid for 3 days with multiple shutdowns, and restarts. It is a Windows 8.1 PC in our child domain. We use Group Policy to control Windows Updates on Windows 7 PCs in both domain, and Windows 8.1 PCs in the parent domain. For all Windows 8.1 PCs in the child domain, we allow Deep Freeze to control Windows Update. I also know that the installation of Deep Freeze disables Windows Fast Startup in Windows 8.1. So while it may be another dead end, I feel like our issue may have something to do with Windows Update policy/services/connectivity, or perhaps Windows Fast Startup (though I don't believe that existed in Windows 7).

 

Would love to know if anyone else has been experiencing long boot delays for the past few weeks. I have done some searching to see if a faulty Windows Update might be to blame, but that doesn't seem likely.

 

Last thing to mention, on a Windows 8.1 system that was having this issue, I uninstalled System Center Endpoint Protection three days ago and it has been fine ever since. The version of Endpoint that was uninstalled was from our post install task (v4.9.219). Through System Center default client policy, it ended up getting Endpoint Protection client v4.7 but it appears to be broken in some way (can't get definitions). We are currently downloading definition updates manually to see if that fixes the issue on that client and then will test shutdowns/restarts to see if the issue comes back. This one may just have other issues in general though since it had been in production for a number of years.

[jslate1980]

Have you limited cpu usage when running scans? What's your scanning schedule?

 

Edit:

How is your update deployment schedule setup?

Have you found and deployment errors in client logs?

Edited March 22, 2017 by jslate1980

[mullet_man]

When you say slow boot times at which point are you seeing an issue?

 

If you have verbose messaging turned on you should be able to work out at which point during the boot process of Windows you are seeing slowness.

[disk]

When you say slow boot times at which point are you seeing an issue?

 

If you have verbose messaging turned on you should be able to work out at which point during the boot process of Windows you are seeing slowness.

 

With verbose messaging, the PCs are stuck on applying user policy. Since there were no changes to policy in either domain, and because of an svchost showing up in every affected PC when using Windows Performance Recorder/Analyzer (from the ADK), which appears to be related to the Microsoft Anti-Malware engine, I'm inclined to believe it is monopolizing the system until it times out and then the system is allowed to finish applying user policy.

[mullet_man]

So this is during user logon?

 

Have you tested on the 2nd logon? Is it just as slow?

 

Are you using roaming/local profiles?

 

Have you tried to clear the profile off the machine and login?

[disk]

It appears as though we may have found the issue! Endpoint Protection seems to have started flagging the Dell Kace agent on our machines. Adding exceptions into the Endpoint client policy has corrected the issue in our initial testing. Waiting for policy to finish propagating to the rest of our PCs.