Major Issues - Server randomly freezing

[Pyroman]

aaaaaaaaaaaaaaaand we're back to the printers falling over but they definitely have the correct config so this must be a server slow down issue

 

The server is back to being incredibly slow, only thing that runs overnight is Redstor backup and Veeam backup.

Edited May 20, 2021 by Pyroman

[Pyroman]

Had stopped sophos on one server for a bit, didn't seem to make a difference.

 

Do you use either splashtop or datto by any chance? I've just noticed that the date of install of the latest update correlates exactly the day when the servers started experiencing issues. Might just be a coincidence but I've removed it from one server and following a reboot it's ran fine for the last half hour.

 

Do you have Windows Defender installed as a feature on those servers?

 

 

When I came in this morning, Windows Defender was mid way through a scan, I stopped that and the server immediately stopped hanging. I've just uninstalled the Feature in Roles & Features and I'm just waiting for a good time to restart.

Edited May 20, 2021 by Pyroman

[Leemanator]

Do you have Windows Defender installed as a feature on those servers?

 

 

When I came in this morning, Windows Defender was mid way through a scan, I stopped that and the server immediately stopped hanging. I've just uninstalled the Feature in Roles & Features and I'm just waiting for a good time to restart.

 

We do, not running real time scan though.

 

I think tonight I'll try removing both that and Sophos as a test on one of the servers and see what happens, tricky thing is that sometimes after a reboot the issue sometimes disappears for a bit.

 

What do you use for backups? I'm looking into the possibility of it being something to do with VSS as there's a few event logs, could be effect rather than cause, but I've reached the stage where I'm throwing stuff against the wall and seeing what sticks!

[TwistedHelixis]

Do you have any of the Windows built in virus or threat systems enabled. Simply having these enabled caused issues with some windows updates on a new server I was setting up recently. The same server was also experiencing odd slow down issues etc until I disabled these.

[Pyroman]

We do, not running real time scan though.

 

I think tonight I'll try removing both that and Sophos as a test on one of the servers and see what happens, tricky thing is that sometimes after a reboot the issue sometimes disappears for a bit.

 

What do you use for backups? I'm looking into the possibility of it being something to do with VSS as there's a few event logs, could be effect rather than cause, but I've reached the stage where I'm throwing stuff against the wall and seeing what sticks!

 

We were doing alright until 1pm where everything has completely bogged down but I haven't had a chance to restart yet. To be fair, I've been trying to get cmd up for the last 15 minutes but the start menu keeps freezing so I can't open it to do a restart!!

 

My Shadow copy routine starts for a local mid-day restore point for files at 1pm which might be a correlation for the server slowdown. I haven't confirmed that's a cause or symptom but it's worth noting.

 

We use Veeam to do a bare-metal full backup every night to external USB and use Redstor to do a 'critial files offsite' backup every night.

[ticktock]

Bit of a long shot but have you checked for loops in the local network. I've seen loops in schools that have sent servers nuts. If you are onsite can you log in to the server with mouse, screen and keyboard then pop the network cable out for a short time to see if the server starts to respond.

[Leemanator]

We were doing alright until 1pm where everything has completely bogged down but I haven't had a chance to restart yet. To be fair, I've been trying to get cmd up for the last 15 minutes but the start menu keeps freezing so I can't open it to do a restart!!

 

My Shadow copy routine starts for a local mid-day restore point for files at 1pm which might be a correlation for the server slowdown. I haven't confirmed that's a cause or symptom but it's worth noting.

 

We use Veeam to do a bare-metal full backup every night to external USB and use Redstor to do a 'critial files offsite' backup every night.

 

Hmm we're just using inbuilt windows backup, there was mention of server backup in the April cumulative release notes which has made me think maybe something to do with this, but doesn't correlate if you're using veeam.

 

I have also found that server slow downs do sometimes happen at the time shadow copies are being taken, but again could be a symptom rather than a cause.

 

It is worth noting that we have a number of servers with the exact same hardware specs, using the same h330 perc controller, that aren't exhibiting any symptoms at all.

[Primus]

I have to say this sounds like failing hardware - possibly drives to me.

[mavhc]

Eons ago the APC UPS software caused server slowdowns due to an expired certificate.

 

You can't see anything odd in task manager, event log, performance monitor?

[Pyroman]

Hmm we're just using inbuilt windows backup, there was mention of server backup in the April cumulative release notes which has made me think maybe something to do with this, but doesn't correlate if you're using veeam.

 

I have also found that server slow downs do sometimes happen at the time shadow copies are being taken, but again could be a symptom rather than a cause.

 

It is worth noting that we have a number of servers with the exact same hardware specs, using the same h330 perc controller, that aren't exhibiting any symptoms at all.

 

Do the other server with no symptoms have Defender installed as a feature even if it's off?

 

From the reddit thread, can you see if this folder is absolutely chocker on both the affected and non-affected servers?

 

----Issue----

Large files generated inside the folder c:/programdata/microsoft/windowsdefender/scan/history

 

----Clarification----

 

This issue is current a known issue and the fix will be hit all release ring in this Thursday. The RCA is Engineer of MsMpEng.dll has some issue and causing lots of files in this folder. The affected engine version is 18100.5.

[Pyroman]

Eons ago the APC UPS software caused server slowdowns due to an expired certificate.

 

You can't see anything odd in task manager, event log, performance monitor?

 

Nothing out of the ordinary apart from what has been reported in this thread....

 

svchost (1212) SoftwareUsageMetrics-Svc: A request to write to the file "C:\Windows\system32\LogFiles\Sum\Svctmp.log" at offset 1966080 (0x00000000001e0000) for 393216 (0x00060000) bytes succeeded, but took an abnormally long time (62 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

 

svchost (1212) SoftwareUsageMetrics-Svc: A request to write to the file "C:\Windows\system32\LogFiles\Sum\Svctmp.log" at offset 3145728 (0x0000000000300000) for 393216 (0x00060000) bytes succeeded, but took an abnormally long time (763 seconds) to be serviced by the OS. This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem.

[mavhc]

And it's Server 2016, not using a VM?

[Pyroman]

Yeah straight Server 2016

[Pyroman]

Bit of a long shot but have you checked for loops in the local network. I've seen loops in schools that have sent servers nuts. If you are onsite can you log in to the server with mouse, screen and keyboard then pop the network cable out for a short time to see if the server starts to respond.

 

I'd be very surprised if Lee and I are both suffering loopbacks but I'll give anything a shot at the moment.

 

When the server locks up I can't even use it locally with Mouse & KB until it decides to wake up again.

[mavhc]

How big is the OS drive/partition compared to any data partitions? Could you clone it to an ssd and boot from sata?

[Leemanator]

I'd be very surprised if Lee and I are both suffering loopbacks but I'll give anything a shot at the moment.

 

When the server locks up I can't even use it locally with Mouse & KB until it decides to wake up again.

 

First thing I tried was taking the network cable out to test this, so yeah not the cause for me at least.

 

 

Think you might be onto something with windows defender, c:\programdata\microsoft\windows defender\history\store has millions of small 2kb files (2m and counting) that's not right!!

 

 

Doing more research now, can you share the reddit thread you mentioned?

 

 

First google results point to this so seems to be a common thread of sophos and windows defender:

https://docs.microsoft.com/en-us/answers/questions/378578/windows-defender-creating-thousands-of-files.html

[Pyroman]

First thing I tried was taking the network cable out to test this, so yeah not the cause for me at least.

 

 

Think you might be onto something with windows defender, c:\programdata\microsoft\windows defender\history\store has millions of small 2kb files (2m and counting) that's not right!!

 

 

Doing more research now, can you share the reddit thread you mentioned?

 

 

First google results point to this so seems to be a common thread of sophos and windows defender:

https://docs.microsoft.com/en-us/answers/questions/378578/windows-defender-creating-thousands-of-files.html

 

Reddit thread is here:

 

Related Microsoft posts are here: https://docs.microsoft.com/en-us/answers/questions/376490/windows-defender-issue-on-server-lots-of-files-bei.html?childToView=381018#answer-381018

and here: https://docs.microsoft.com/en-us/answers/questions/378578/windows-defender-creating-thousands-of-files.html

Edited May 20, 2021 by Pyroman

[Leemanator]

Reddit thread is here:

 

Related Microsoft posts are here: https://docs.microsoft.com/en-us/answers/questions/376490/windows-defender-issue-on-server-lots-of-files-bei.html?childToView=381018#answer-381018

and here: https://docs.microsoft.com/en-us/answers/questions/378578/windows-defender-creating-thousands-of-files.html

Thanks, looking into it I suspect the fix lies not in stopping defender, as Microsoft seems to have fixed the it and is no longer creating new files, but in deleting the millions of files it's already created. I'm running the below in powershell to clear the files off (can do without the 'select-first 100000', but I think this would cause it to crash if you tried to delete them all at once) as I can't even load the folder in explorer with that many files.

 

 

[iO.Directory]::EnumerateFiles("C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store\") | select -first 100000 | remove-item -Verbose

[ronnoco]

Hi,

 

Not sure if this has already been mentioned but I would recommend using perform to monitor the disks and see if you are suffering any latency. Collect over a period of time. I once had a SAN that was running like treacle and using perfmon showed massive latency. For my scenario, updating the NIC drivers to the SAN switch resolved the issue. Could be worth checking your NIC drivers. Also, you could run perfmon on the NIC's as well.

 

Here is a quick guide on how to use it if you've not come across it before.

 

https://www.smartertools.com/blog/2016/07/15-configure-perfmon-to-prevent-disk-issues

 

Also, I would suggest uninstalling any AV...just to eliminate. Sophos has caused me all sorts of issues in the past.

 

Hope you get it fixed.

[Pyroman]

Thanks, looking into it I suspect the fix lies not in stopping defender, as Microsoft seems to have fixed the it and is no longer creating new files, but in deleting the millions of files it's already created. I'm running the below in powershell to clear the files off (can do without the 'select-first 100000', but I think this would cause it to crash if you tried to delete them all at once) as I can't even load the folder in explorer with that many files.

 

 

[iO.Directory]::EnumerateFiles("C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store\") | select -first 100000 | remove-item -Verbose

 

Thanks for the script I've upped it to 5,000,000 and it's working. slowly. See how we look tomorrow when its run overnight.

[Pyroman]

10GB of files gone in 16 hours of running the script, no idea how much more to go because it crashed every time I tried to see how many files there were in that folder! Looks like it's gonna have to run all weekend.

[Leemanator]

Yeah still deleting on most of my servers too, 16 million 2kb files on one of my servers!! definitely one to leave running over the weekend.

[Pyroman]

Yeah still deleting on most of my servers too, 16 million 2kb files on one of my servers!! definitely one to leave running over the weekend.

 

If you struggle with speed, this tool cleared stuff really quickly:

 

https://github.com/McNetic/fdeltree

 

Not sure if it was any quicker than the Powershell command but I've managed to totally remove the folder now so hoping everything will come back up to full speed.

 

Weirdly Windows updates were broken again Ran the troubleshooter and it fixed "service registration is missing or corrupt" error. It's now trying to download KB5001402 which should have been done in April so not sure why that broke.