Hyper-V Cluster CSV Performance Slow!

[superaz300]

Hey Guys,

 

Just setup a Server 2019 Hyper-V LAB, with 2 nodes. The nodes have an iSCSI connection to a Dell MD3200i SAN, see this thread here regarding the setup: http://www.edugeek.net/forums/hardware/206389-home-lab-hyper-v-san-setup-advice.html

 

When I copy something to the iSCSI Disk from the node, it was okay and I was getting around 230MB. Now, since turning this into a failover cluster, and adding the same iSCSI disk as a CSV, the performance is terrible! As a test, I copied a 4GB ISO to C:\ClusterStorage\Volume 1 and it copied at around 22MB, if I take the disk out of the cluster and copy the same file directly to the iSCSI disk, I get around 230MB which is what I would expect. (I think). I know some are going to shoot me down and say I shouldn't copy stuff to the CSV folder, so I created a VM on the CSV disk, the same speeds occure inside the VM, I get around 25MB.

 

What could cause this issue? I can confirm the cluster is full patched to the latest level.

 

Best regards

James

[ChrisMiles]

Didn't format it with ReFS did you? CSV volumes can't work properly with ReFS, at least on Server 2016, they go into redirected access mode.

[superaz300]

Hey Chris,

 

Thank you for the reply. Ohhh, hmm I will check that, I take it I will find this on the properties of the partition?

 

Thank you

James

 

Didn't format it with ReFS did you? CSV volumes can't work properly with ReFS, at least on Server 2016, they go into redirected access mode.

[ChrisMiles]

Hey Chris,

 

Thank you for the reply. Ohhh, hmm I will check that, I take it I will find this on the properties of the partition?

 

Thank you

James

 

Yeah, it would either be NTFS or ReFS.

[superaz300]

Hi Chris,

 

From what I can see, it is only telling me the drive is CSVFS. If I take the drive out of the cluster, it does show as NTFS.

 

Best regards

James

 

Yeah, it would either be NTFS or ReFS.

[ChrisMiles]

Hi Chris,

 

From what I can see, it is only telling me the drive is CSVFS. If I take the drive out of the cluster, it does show as NTFS.

 

Best regards

James

 

Not that then. Might be something to do with MPIO since you're using iSCSI. How many paths do you have to the NAS for each host? Try removing all but 1 per host and see if things chance.

[superaz300]

Hi Chris,

 

It could be MPIO, I have never setup a SAN before so this was my first time, hence it been a lab. This is my configuration physically:

SAN Controller 0 Port 0 -> Switch 1, Port 1 (192.168.130.101) VLAN 130

SAN Controller 0 Port 1 -> Switch 2, Port 1 (192.168.131.101) VLAN 131

SAN Controller 0 Port 2 -> Switch 1, Port 2 (192.168.132.101) VLAN 132

SAN Controller 0 Port 3 -> Switch 2, Port 2 (192.168.133.101) VLAN 133

 

SAN Controller 1 Port 0 -> Switch 1, Port 3 (192.168.130.102) VLAN 130

SAN Controller 1 Port 1 -> Switch 2, Port 3 (192.168.131.102) VLAN 131

SAN Controller 1 Port 2 -> Switch 1, Port 4 (192.168.132.102) VLAN 132

SAN Controller 1 Port 3 -> Switch 2, Port 4 )192.168.133.102) VLAN 133

 

Switch 1 Port 5 -> Host 1, iSCSI NIC 1 (192.168.130.201) VLAN 130

Switch 2 Port 5 -> Host 1, iSCSI NIC 2 (192.168.131.201) VLAN 131

Switch 1 Port 6 -> Host 1, iSCSI NIC 3 (192.168.132.201) VLAN 132

Switch 2 Port 6 -> Host 1, iSCSI NIC 4 (192.168.133.201) VLAN 133

 

Switch 1 Port 7 -> Host 2, iSCSI NIC 1 (192.168.130.202) VLAN 130

Switch 2 Port 7 -> Host 2, iSCSI NIC 2 (192.168.131.202) VLAN 131

Switch 1 Port 8 -> Host 2, iSCSI NIC 3 (192.168.132.202) VLAN 132

Switch 2 Port 8 -> Host 2, iSCSI NIC 4 (192.168.133.202) VLAN 133

 

I am not saying this is correct at all, this is just information I have gathered from the Deployment Guide of the MD3200i and also advice from people. In terms of MPIO, I used the Dell Disk Manager to automatically create the iSCSI connections and MPIO settings using the recommended configuration.

 

Thank you

James

 

 

Not that then. Might be something to do with MPIO since you're using iSCSI. How many paths do you have to the NAS for each host? Try removing all but 1 per host and see if things chance.

[ChrisMiles]

That doesn't look wrong necessarily, but some iSCSI SANs can be really weird. The first thing I would do is disconnect from the switch all but the VLAN 130 cables (1 for each controller, 1 for each host) without otherwise changing the configuration and see if there is any chance in performance. Edited May 13, 2019 by ChrisMiles

[superaz300]

Hi Chris,

 

Thank you again for the reply. I will indeed try this. I do have doubts about my switches, they are very old HP-180024G switches, which are cheap and nasty but they are all I have in my lab. I do plan in the future to get proper iSCSI switches, but if having a direct connection fixes this issue, I will certainly make do with that.

 

Best regards

James

 

 

 

That doesn't look wrong necessarily, but some iSCSI SANs can be really weird. The first thing I would do is disconnect from the switch all but the VLAN 130 cables (1 for each controller, 1 for each host) without otherwise changing the configuration and see if there is any chance in performance.

[ChrisMiles]

One other thing - I know that even some older enterprise switches (HP at least) can't do jumbo frames and flow control at the same time so if you've enabled these perhaps they might be causing issues.

[superaz300]

Hi Chris,

 

That is also interesting, because I have Flow Control enabled and Jumbo Frames at 9K. I wonder if this is worth turning off?

 

James

 

One other thing - I know that even some older enterprise switches (HP at least) can't do jumbo frames and flow control at the same time so if you've enabled these perhaps they might be causing issues.

[ChrisMiles]

Hi Chris,

 

That is also interesting, because I have Flow Control enabled and Jumbo Frames at 9K. I wonder if this is worth turning off?

 

James

 

Try the single cables thing first and if that doesn't help maybe try turning off jumbo frames, the performance gain is tiny to nil anyway. I assumed you enabled it on the SAN, switches and hosts?

[superaz300]

Hi Chris,

 

Yes, I enabled Jumbo Frames on the SAN, Switches and Hosts. I confirmed I could ping the san from the hosts using ping -f -l 6000 192.168.130.101. I also enabled Flow Control on the switches, and set the adaptors on Window to TX and RX Enabled, couldn't see an option for Flow Control on the SAN. Also, on the switches I cannot specify a packet size, I can only enable Jumbo Frames itself.

 

I will try what you have suggested tonight.

 

Best regards

James

 

Try the single cables thing first and if that doesn't help maybe try turning off jumbo frames, the performance gain is tiny to nil anyway. I assumed you enabled it on the SAN, switches and hosts?

[superaz300]

Try the single cables thing first and if that doesn't help maybe try turning off jumbo frames, the performance gain is tiny to nil anyway. I assumed you enabled it on the SAN, switches and hosts?

 

Hi Chris,

 

Tried what you have said, and sadly it is still the same. Tried brand new CAT 6 cables as well, still the same speed. This is really frustrating. As a test, I have created a virtual disk on the SAN and mapped it to the node just as a normal hard disk in iSCSI, copying the same file copies over 210MB, copy it to the CSV, 22 - 25MB.

 

Any more ideas?

 

Thank you

James

[superaz300]

I should also say I have disabled Jumbo Frames on the SAN and Nodes, everything is now direct removing the switches from the situation.

[superaz300]

 

This is interesting, just tried a copy 4GB file inside the VM which is located on the CSV. It started off really good, in fact it was copying at about 270MB, got about 25% of the way then quickly reduced to 22MB copy. Could this be some kind of caching issue some where?

[PotNoodleTech]

[ATTACH=CONFIG]53351[/ATTACH]

 

This is interesting, just tried a copy 4GB file inside the VM which is located on the CSV. It started off really good, in fact it was copying at about 270MB, got about 25% of the way then quickly reduced to 22MB copy. Could this be some kind of caching issue some where?

 

Yes it could be how much battery backup cache you have on your raid card?

[ChrisMiles]

Hmmmmm OK, when its clustered is the host you are doing the copy from the owner of the disk? Also check for redirected access mode anyway by running Get-ClusterSharedVolumeState on the cluster node. Post the result if you want.

[superaz300]

Hmmmmm OK, when its clustered is the host you are doing the copy from the owner of the disk? Also check for redirected access mode anyway by running Get-ClusterSharedVolumeState on the cluster node. Post the result if you want.

 

Hi Chris,

 

Thank you again for the reply. I can confirm the host I am doing this on is the owner of the disk, I have also tried this on the other host in the cluster and I get exactly the same result. Here is the output from the above command.

 

BlockRedirectedIOReason : NotBlockRedirected

FileSystemRedirectedIOReason : NotFileSystemRedirected

Name : Cluster Disk 2

Node : JE-NODE-01

StateInfo : Direct

VolumeFriendlyName : Volume1

VolumeName : \\?\Volume{4cf6f351-82a5-4af0-bbe1-135d68c7d927}\

 

BlockRedirectedIOReason : NotBlockRedirected

FileSystemRedirectedIOReason : NotFileSystemRedirected

Name : Cluster Disk 2

Node : JE-NODE-02

StateInfo : Direct

VolumeFriendlyName : Volume1

VolumeName : \\?\Volume{4cf6f351-82a5-4af0-bbe1-135d68c7d927}\

 

Best regards

James

[ChrisMiles]

Well that looks normal. I have to admit I'm running out of ideas! Any chance your fast out-of-cluster tests are just a result of IO caching on the host and or SAN? You could try running some random IO tests. 22 MB/s is not very fast at all though.I was reading the details of your lab. What setup did you go with for the disk config in the SAN in the end? 7.2k sata disks are not going to do much in the way of iops and if you chose raid 5 or even 6 it may be that simply the san controller is too slow to cope. I am just guessing now though.

[ChrisMiles]

Couple of things to try here:

https://www.dell.com/community/PowerVault/Very-slow-READ-performance-with-MD3200i/td-p/3577960/page/2

Including disabling RSS and TCp Chiney Offload on the host nics (I think youre meant to do this anyway for iscsi) and disabling flow control.

[Oaktech]

Insufficiently deep packet buffer on your switches?

 

Ye Olde Broadcom network driver issue?

[superaz300]

Hi guys,

 

Thank you again for the replies.

 

First of all, I decided to go for RAID 10 in the end with all 12 disks. I am no longer using the switches, the SAN is now directly connected to the hosts to outrule the switches been an isuse. It is interesting this post would Chris posted: https://www.dell.com/community/PowerVault/Very-slow-READ-performance-with-MD3200i/td-p/3577960/page/2

 

The adaptors I am using are exactly the ones mentioned here, BCM5709C. However, this command was not recognised when I ran it: netsh int tcp set global chimney=disabled

 

I did run the other command though, sadly this hasn't made no difference what so ever.

 

Just as a thought, I read a post the other night about been able to manage the storage network "iscsi" within the Failover Cluster Manager. However, I never created my storage network through Hyper-V, I created them directly on the hosts before I even installed the Hyper-V role, could this maybe have anything to do with this issue. Should I delete the storage network configuration, and create 2 new iSCSI networks using the Virtual Switch Manager?

 

These replies and the help I am getting is fantastic, thank you so much.

 

Best regards

James

[ChrisMiles]

Try this one also:

 

netsh int tcp set global autotuning=disabled

 

IIRC this is actually the main problem one, so try this first. Requires a reboot.

 

https://www.dell.com/support/article/us/en/04/sln132130/windows-server-disabling-tcp-chimney-offloading-and-autotuning-may-improve-iscsi-performance?lang=en

 

Don't worry about the Hyper-V vs Cluster manager thing, all cluster manager does is ensure hosts have the same settings and if this wasn't the case it would have shown on the cluster report.

That does make me think of something though, the iscsi networks should show in cluster manager and should be set to not be used by the cluster at all. They should not have cluster comms enabled.

Also, on your iscsi network adapters did you disable all the adapter bindings except TCPIP4? And NetBIOS over TCP/IP.

Edited May 15, 2019 by ChrisMiles

[superaz300]

Try this one also:

 

netsh int tcp set global autotuning=disabled

 

IIRC this is actually the main problem one, so try this first. Requires a reboot.

 

https://www.dell.com/support/article/us/en/04/sln132130/windows-server-disabling-tcp-chimney-offloading-and-autotuning-may-improve-iscsi-performance?lang=en

 

Don't worry about the Hyper-V vs Cluster manager thing, all cluster manager does is ensure hosts have the same settings and if this wasn't the case it would have shown on the cluster report.

That does make me think of something though, the iscsi networks should show in cluster manager and should be set to not be used by the cluster at all. They should not have cluster comms enabled.

Also, on your iscsi network adapters did you disable all the adapter bindings except TCPIP4? And NetBIOS over TCP/IP.

 

Hi Chris,

 

Thank you for this, just applied that command and rebooting the host now.

 

Ah, well its interesting because the iSCSI network doesn't show at all in the Cluster Manager on both hosts. I never disabled the bindings you mentioned either, I will also do that.

 

Best regards

James