Leemanator

Yeah still deleting on most of my servers too, 16 million 2kb files on one of my servers!! definitely one to leave running over the weekend.

Thanks, looking into it I suspect the fix lies not in stopping defender, as Microsoft seems to have fixed the it and is no longer creating new files, but in deleting the millions of files it's already created. I'm running the below in powershell to clear the files off (can do without the 'select-first 100000', but I think this would cause it to crash if you tried to delete them all at once) as I can't even load the folder in explorer with that many files. [iO.Directory]::EnumerateFiles("C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store\") | select -first 100000 | remove-item -Verbose

First thing I tried was taking the network cable out to test this, so yeah not the cause for me at least. Think you might be onto something with windows defender, c:\programdata\microsoft\windows defender\history\store has millions of small 2kb files (2m and counting) that's not right!! Doing more research now, can you share the reddit thread you mentioned? First google results point to this so seems to be a common thread of sophos and windows defender: https://docs.microsoft.com/en-us/answers/questions/378578/windows-defender-creating-thousands-of-files.html

Hmm we're just using inbuilt windows backup, there was mention of server backup in the April cumulative release notes which has made me think maybe something to do with this, but doesn't correlate if you're using veeam. I have also found that server slow downs do sometimes happen at the time shadow copies are being taken, but again could be a symptom rather than a cause. It is worth noting that we have a number of servers with the exact same hardware specs, using the same h330 perc controller, that aren't exhibiting any symptoms at all.

We do, not running real time scan though. I think tonight I'll try removing both that and Sophos as a test on one of the servers and see what happens, tricky thing is that sometimes after a reboot the issue sometimes disappears for a bit. What do you use for backups? I'm looking into the possibility of it being something to do with VSS as there's a few event logs, could be effect rather than cause, but I've reached the stage where I'm throwing stuff against the wall and seeing what sticks!

Had stopped sophos on one server for a bit, didn't seem to make a difference. Do you use either splashtop or datto by any chance? I've just noticed that the date of install of the latest update correlates exactly the day when the servers started experiencing issues. Might just be a coincidence but I've removed it from one server and following a reboot it's ran fine for the last half hour.

Yeah we've updated the bios/raid controller firmware to latest and still seeing issues, I'm starting to bang my head against the wall as well. All our affected servers are using the H330 Perc adapter, I suspect it's due to this controller having no cache. Are you using raid 5/6? One of the servers that was affected worst by it yesterday morning has gone back to normality after a reboot, and been fine now for the best part yesterday and this morning, whereas a few others have gotten worse and reboots did not help. What A/V you running? we've got Sophos Central just wondering if that correlates with you? disabling the real time scan didn't seem to resolve any issues, but it's the only third party software that's installed on all affected servers.

We're in the process of running the SUU on the affected servers now, will let you know how it goes for us. I've found rebooting the server seems to alleviate the issue for a short period so we've managed to avoid any end user issues by rebooting when needed. All affected servers have been updated up to the April cumulative update (kb5501347), but it doesn't seem to correlate with when the issues occurred, some servers had this installed for several weeks before any faults started happening, and one server seemed to be having this issue before this update was applied looking into event viewer, so sort of suggests it wasn't this update at fault... we've started deploying the may cumulative to see if this has a fix on it but not hopeful Also we've had this issue on a couple of server 2019 boxes too so doesn't seem to be 2016 specific. Hopefully the bios & firmware updates will work, we are a few versions behind by the looks.

Are you getting anything in the event viewer like the below? Log Name: Application Source: ESENT Date: 17/05/2021 12:50:38 Event ID: 533 Task Category: General Level: Warning Keywords: Classic User: N/A Computer: ******* Description:svchost (3364,T,0) SRUJet: A request to write to the file "C:\WINDOWS\system32\SRU\SRU.chk" at offset 0 (0x0000000000000000) for 4096 (0x00001000) bytes has not completed for 36 second(s). This problem is likely due to faulty hardware. Please contact your hardware vendor for further assistance diagnosing the problem. We've had a handful of Dell Poweredge Servers start freezing up randomly in the past couple of weeks, with event's like these showing writes are taking a long time. No hardware errors in open manage or when running diagnostics. Suspect it's to do with the April cumulative but can't see a pattern between the servers affected and the servers unaffected as yet.