XenCenter Troubleshooting (newbie) after mass server outage

[Asreal]

Hi all, resident newbie here ashamedly only logging in when I want something from the wise-folk, I hope you none of you are offended!

 

Okay the situation:

After experiencing the usual horrors that can only occur during the first few weeks of September, things had (finally) started to calm down to and almost pleasant "mildly annoying" level.

Then, we reach this afternoon and each of our Windows Servers seemingly just grind to a halt and die one by one.

 

There didn't seem to be any particular order to this; we get initial reports of the Internet being a bit slower than normal (during the morning), then we lose Internet connectivity altogether, file server starts to die (still pingable, but not able to remote in), followed by the rest (DC, DHCP etc etc).

 

We had had this problem once before, I'd tried:

 

* testing for broadcast storms:

- We have a switch that has a handy little LED display that shows network utilisation, we plug this into our core switch and begin unplugging fibre modules to see if we can narrow down cabinets with high sources of traffic - no luck

 

* testing extrernal access

- plugged a laptop (static IP/GW) directly into our router could access T'Internet fine

 

* rebooting virtual servers:

- tried doing this via a remote session only to find that all my existing remote sessions (open near permanently) had died

- tried accessing the console in XenCenter itself only to find that it was barely responsive and would not let me in

- initially did a (nice) shutdown on said servers to try and minimise any damage

- afer waiting 30 mins, gave up and did a force shut down (which worked quite quickly)

- Linux (web) servers seemed unaffected and would happily let me shut them down from the console without any problems

 

Only shutting everything down and rebooting our hosts and slowly bringing everything else back up seemed to work. This I had to do today, but don't relish the idea of doing it again...

 

 

* possibilities

- hardware is fairly new (hosts: Dell Poweredge R710's SAN: Dell PowerVault MD6300i), but we only have ONE iSCSI port plugged in

(this is how I inhereted it, I want to get the other plugged in but don't know enough about this yet to try - thet "if it works/worked don't change it (yet)" mentality etc. I understand that it has worked like this for ages...)

- we are running XenCenter 6.0.2 and due to update it soon, but I can't see as this would affect anything that majorly (not after we've already been using it for so long)

 

Any ideas of where I should start?

 

Thanks in advance!!