XenCenter Troubleshooting (newbie) after mass server outage

[Asreal]

Hi all, resident newbie here ashamedly only logging in when I want something from the wise-folk, I hope you none of you are offended!

 

Okay the situation:

After experiencing the usual horrors that can only occur during the first few weeks of September, things had (finally) started to calm down to an almost pleasant "mildly annoying" level. Then, we reach this afternoon and each of our Windows Servers seemingly just grind to a halt and die one by one.

 

There didn't seem to be any particular order to this; we get initial reports of the Internet being a bit slower than normal (during the morning), then we lose Internet connectivity altogether, file server starts to die (still pingable, but not able to remote in), followed by the rest (DC, DHCP etc etc).

 

We had had this problem once before, I'd tried:

 

* testing for broadcast storms:

- We have a switch that has a handy little LED display that shows network utilisation, we plug this into our core switch and begin unplugging fibre modules to see if we can narrow down cabinets with high sources of traffic - no luck

 

* testing extrernal access

- plugged a laptop (static IP/GW) directly into our router could access T'Internet fine

 

* rebooting virtual servers:

- tried doing this via a remote session only to find that all my existing remote sessions (open near permanently) had died

- tried accessing the console in XenCenter itself only to find that it was barely responsive and would not let me in

- initially did a (nice) shutdown on said servers to try and minimise any damage

- afer waiting 30 mins, gave up and did a force shut down (which worked quite quickly)

- Linux (web) servers seemed unaffected and would happily let me shut them down from the console without any problems

 

Only shutting everything down and rebooting our hosts and slowly bringing everything else back up seemed to work. This I had to do today, but don't relish the idea of doing it again...

 

 

* possibilities

- hardware is fairly new (hosts: Dell Poweredge R710's SAN: Dell PowerVault MD6300i), but we only have ONE iSCSI port plugged in

(this is how I inhereted it, I want to get the other plugged in but don't know enough about this yet to try - thet "if it works/worked don't change it (yet)" mentality etc. I understand that it has worked like this for ages...)

- we are running XenCenter 6.0.2 and due to update it soon, but I can't see as this would affect anything that majorly (not after we've already been using it for so long)

 

Any ideas of where I should start?

 

Thanks in advance!!

[dhicks]

Any ideas of where I should start?

 

We run Xen, on very similar hardware to you (Dell R510 / R710 servers). Is your iSCSI device attached to the same network as the rest of the machines in the school? If so, could someone have simply plugged in a machine with the same IP address as your SAN device - maybe a home laptop configured with a static IP address? I did have an issue a few years back with a corrupted XenServer Storage Repository (SR) metadata partition, but if you've managed to reboot your machines okay I'm guessing that's not the issue.

[dhicks]

Just to add: you know there's a new security exploit and associated fix out for XenServer? Amazon and Rackspace having been patching their systems recently, it might be a good idea to get your systems patched now, too.

[Asreal]

Hi Mr Hicks, thanks for responding so quickly!

 

iSCSI: Yes it's attached to our main core switch along with *everything* else on the network, but my predassessor had put it on a 192.168.x network rather than what the rest of the school are on.

(It could be someone bringing something in from home that defaults to a 192.168 address? That's actually something I'm going to look at now that you mention it...)

 

Patching: Is it part of a Xen update or seperate patch?

 

While you're on, do you know if there are any network monitoring tools/logs that are part of the XenCenter console?

 

Thanks again for getting back to me - at least I've got a direction to start in.

[dhicks]

Patching: Is it part of a Xen update or seperate patch?

 

It should just be one of the update notifications that appears in XenCenter - we have one dated 1st October, I imafgine that's the patch that Amazon and so forth have been applying recently.

 

While you're on, do you know if there are any network monitoring tools/logs that are part of the XenCenter console?

 

Our XenServer setup is one of those things I intended to get all sorted and running with monitoring and so forth, but simply haven't had the time. I would imagine that an SNMP agent would be available somewhere for XenServer, although it might wind up being one of those components you have to compile and add in as a module.

[Achandler]

Your iSCSI IPs should defeinitely be on an entirely different range then the main network, using 192.168 is absolutely fine if your switches are VLANed off properly. Don't go changing them because the connections will fail.

 

Our iSCSi for our Xen servers are on 192.168 but you can't ping them or go near them because they are VLANed off seperately.

[Sephiroth]

I used to run XenServer, and did have some problems with it. I seem to recall finding something on one of the log files, though I can't quite remember which one. they should be in "/var/log/messages" or "/var/log/xensource.log"

 

I believe that these are the main log files for XenServer.