So it was always my impression that one of the benefits of virtualization was less disasters and easier recovery. And practically the only way to loose a virtual server would be either accidently broke windows or the SAN broke. In any other circumstance the VM could be recovered from the SAN. We have a backup NAS for the most important VMs as a means for fast recovery in case of accidental breakage. Important Vms like print server or Domain controller. Other things It wouldn't be disaster to have to reinstall windows.

In the previous non virtualized system we had 5 main servers, a san, and a bunch of other servers doing various jobs. If one of the 5 main servers went down we'd have to attach the volume on the san to one of the other servers and maybe remap a bunch of user areas. Any other server that broke would only affect its singular function. In the 7 year life of those 5 servers we never had a major outage.

In the new system we have 3 host servers running vmware off SD card and a 12TB san hosting all the data. On Thursday night one of the host servers crashed which put a 3rd of the VMs offline. Luckily it was mostly not any important ones.

After rebooting and a day of our support company trying to fix the situation we have 1 print server restored from backup, 6 other servers wiped out and a call in to vmware because VMX files have locks which refuse to delete making the vms unrecoverable!

I really hope Vmware know what to do as otherwise we just lost 1/3 of the VMs...

Imaging server + all the images

Proxy server for computer maintenance

Antivirus sophos server

Remote desktop server

Lucid Server (learning support program)

Dev server (all the install files, ISOs, login logs etc)

I'm shocked that something this bad could happen. Now the NAS is going to have to back up all the VMS just in case VMware kills them all, make sure the DCs are never on the same host, have a duplicate print server on a different host and consider having some functions running outside of the virtual system altogether.