ESX Host has insane throughput to LUNs

[DrPerceptron]

We have 3 ESX hosts, one of which is pushing through insane (50 times more than the other two hosts) throughput and IOPS values to the LUNs it is attached to, however, it doesn't appear to be a specific VM that is causing the issue.

 

The other two hosts push about 1MBp/s average to groups, this one is pushing 60MB+ and latency is periodically rising from an average of 4300ms to 1.1 seconds.

 

Ignoring smoothwall, which is the highest CPU/Disk usage VM on the host currently, the others are very low usage VMs, there aren't any hardware errors appearing and I'm slightly out of ideas what to look for.

[pantscat]

Personally, I found that having Smoothwall as a VM caused unacceptable levels of disk latency on my VM environment.

 

Have you tried vMotion-ing it to another host to see if it causes issues on that one?

[DrPerceptron]

it's in the process of moving at the moment.

[pantscat]

Probably worth changing the default performance reporting to see if any of the VMs are sending lots of write or read requests - although with the latency the way it is currently it may not be terribly obvious as to which the culprit is.

[DrPerceptron]

playing with Foglight at the moment to see if I can come up with anything.

 

I've been moving around some of the VMs that might have influence, but smoothwall likes returning to the crazy host, but isn't really making a big difference to the stats when it is moved away.

 

Since this morning, it's dropped from 60+ to just below 10 which is considerably better than before, but I still haven't been able to figure out why it's so high without any obvious reasoning - even ESXTOP doesn't seem to think much of it, aside from the large volume of traffic flowing to the LUNs from this one host.

 

The other servers have some heavy usage VMs on and are reporting largely expected values - CCTV has an estimated throughput of 5Mbps, the LUN it lives on also has the same throughput, the same for the sum total of our SQL boxes and Exchange server too.

[pantscat]

I've been moving around some of the VMs that might have influence, but smoothwall likes returning to the crazy host

 

Are you running DRS? Why not turn it off and move all VMs off on to the other hosts to see if the LUN spikes still occur? Could be the ESXi box doing something odd.

[xenonive]

We had big increase when we updated the HP drivers via update manager for ESXI , does the good host have newer version of ESX ? do you have server drivers installed ?

[DrPerceptron]

Are you running DRS? Why not turn it off and move all VMs off on to the other hosts to see if the LUN spikes still occur? Could be the ESXi box doing something odd.

 

Yes, When the host is in maintenance mode, it's still pushing about 60 IOPS and 5MB up there for some reason, the other hosts don't have the same problems, but the device latency does go flying up running across just the other two nodes.

 

We had big increase when we updated the HP drivers via update manager for ESXI , does the good host have newer version of ESX ? do you have server drivers installed ?

 

I'm not aware of any driver or OS updates since quite a while ago, we have pushed out the newest 5.1 updates to them to see if that changed anything but it hasn't. 5.5 is on the cards, but not until half-term probably.

 

I checked the versions, but they are the same.