Hyper-V Cluster Performance Problem

[ITGURU]

I have inherited a network setup by an external company consisting of:

1 x Windows Based SAN with 4 x onboard NICs all teamed within Windows,

2 x SAN Switches

2 x Hyper-V hosts, with VMs on both Hosts using CSV storage on the windows SAN. - 4 x NICs connected to the SAN switches, (2 each per host) and 4 NICs teamed to the main network side.

When logging into any of the VMs or performing any management tasks, the performance is so slow in comparison to dedicated SANs i have worked on in the past and normal standalone Hyper-V setups.

How do i determine what the issue is?

The Windows SAN has 2 x SSDs for host OS and a RAID 10 which is running the Storage Volume for the CSV.

If i copy an iso from the Hyper-V host to the data drive on the SAN i get consistent 110+ MB/s transfer speed, however if i copy it to one of the VMs on the cluster storage I get below 40MB/s copy speed and occasionally will stall and drop to 0MB/s.

I'm not sure that something is correctly configured, including MPIO to use all available paths.

[Steve21]

As you said about them being 1Gb the big one to check normally is whether VMQs are still enabled on the NICs as they can cause problems depending on drivers etc

 

Do you get the same kind of speeds if you copy it from an external pc to the VM etc? rather than the HOST?

 

Steve

[ITGURU]

As you said about them being 1Gb the big one to check normally is whether VMQs are still enabled on the NICs as they can cause problems depending on drivers etc

 

Do you get the same kind of speeds if you copy it from an external pc to the VM etc? rather than the HOST?

 

Steve

 

VMQ is disabled.

If copy from external pc to a VM on the CSV it fluctuates all over from 10mb/s to 50mb/s.

 

If i work on any of the VM servers they are slow with latency, whereas if move the vm virtual disk to the local host it performs well.

 

SAN iscsi ports are using 100-200mb/s utilisation of the 1gb links.

 

Working on servers on a SAN/Hyper-V always seem to be slower than on a physical server whether doing updates or day to day use.

[ronnoco]

I would try running perfmon to see if you can identify any latency on the drives or nic's. Remember to run the tests from one of the nodes not the roles for accurate reporting. I had a similar problem in the past which turned out to be network driver related. Inconsistencies in driver versions caused terrible slowness. Updated the nic drivers on all the nodes that connected to the switch fabric and it resolved the issue. Are the nodes up to date? Also, have you got any SAN management software you can use to check the general health of the SAN?

[ITGURU]

Yes, all nodes are up to date.

The 'SAN' is basically a Dell server configured with 1 x large 20TB storage pool in RAID 10, running the 4 onboard ports for iscsi to the hosts!

All inherited, so not something I setup!

[ronnoco]

Oh, I see. Definitely worth running perfmon over a period of time. Monitor the hard drive and nic performance. Have you looked at the hardware logs, event logs, etc on OpenManage?

 

Worst case scenario, could you run the roles standalone across all your servers? That Dell sounds quiet a beast and would probably handle a lot just by itself.

 

I'm not a lover of SAN's myself. Too complicated!