Couple of VMs crashed last night

[Norphy]

I have a bit of a strange one.

 

I have two VMs acting as file servers. They are on different VM clusters in geographically separate sites. The thing that links them is that they're DFS mirrors

 

Last night, both of them rebooted and got stuck on this screen:

 

Pressing the continue button on the servers allowed them to finish booting and they're working once again.

 

The odd thing is, I can't find why this happened. I can see in my monitoring software the time that they stopped responding (around 11pm last night for one, about 2.30 this morning for the other). I've looked in the event logs and I can see a big gap from when they stopped responding but no clue as to why, no unusual event log messages. I don't think they blue screened as Windows is usually quite good about putting in events when it does that. There is one event in the Kernel log complaining about an unclean reboot but that's it.

 

The monitoring software shows nominal CPU and RAM usage just before they crashed. The disk IO looks normal for the time that they crashed as well. Backups were being taken at the time but they completed without issue.

 

Ordinarily I'd just write this off as one of those things but because this server hosts some important shares, my boss wants an explanation as to why it happened. It's weird that two VMs on two VM hosts in different clusters in different building both reboot overnight. Does anyone have any suggestions as to where else I can look?

[bald_pig]

Spitballing ideas here, but A/V trying to scan a file as it's being replicated perhaps?

 

For them both to go down we should assume that it's linked to the DFS, if that's the only thing that distinguishes them from all other servers.

[Norphy]

Thanks

 

Nice idea but AV is disabled on those VMs.

 

Both VMs are part of HyperV clusters, there are events in both of the cluster logs complaining about failed roles from after they crashed but I suspect this is a symptom rather than being indicative of any cause

[jthompson]

I know you mentioned that it didn't look like BSODs, but have you checked for any BSOD dump files in %windir%\minidump ?

[Norphy]

I did check for memory dumps, yes.

 

I've found what happened.

 

I enabled deduplication on these servers yesterday and started an optimization run. That was still underway when the backups started. Veeam took a snapshot, created a AVHDX which eventually filled up the remaining space on the volume that the VHDXs are stored on. This caused the VM to crash. Veeam eventually finishes, merges the AVHDX back in and frees the space back up. The servers restart but get stuck on that screen.

 

Annoying but at least I can see what happened.