ISCSI Switching - Transmitted Paused Frames

[glennda]

I'm no ISCSI expert having mainly worked with Fiber Channel.

 

Today I'm looking through some performance issues with a Dot Hill San and VM's hanging for a couple of seconds every now and then. One of the things I have noticed is there is some Transmitted Pause Frames on the ISCSI Switching (dedicated HP 1810-24G - Jumbo Frames Enabled).

 

Its not a huge amounts of packets considering the numbers of transmitted but I'm not sure if its a factor?

 

Ports 1-4 are trunked to a seperate switch which is seeing pretty similar numbers

 

Are these switches struggling to process the traffic?

 

[SYNACK]

Looks like the answer is yes, it may not be the packet numbers but jumbo frames choking the buffers out:

http://packetpushers.net/flow-control-storm-%E2%80%93-ip-storage-performance-effects/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013413

[glennda]

Looks like the answer is yes, it may not be the packet numbers but jumbo frames choking the buffers out:

http://packetpushers.net/flow-control-storm-%E2%80%93-ip-storage-performance-effects/

VMware KB: Configuring Flow Control on ESX and ESXi

 

Thanks Synack - First link isn't working but will look later and read through the VMware (they are having issues with DDOS according to Status Page.)

[DMcCoy]

Any particular reason for the jumbo frames? I never found it made much difference with 1Gb iSCSI. The 1810 really is a low end switch, even if it is all Gb and fanless.

[glennda]

Any particular reason for the jumbo frames? I never found it made much difference with 1Gb iSCSI. The 1810 really is a low end switch, even if it is all Gb and fanless.

 

I know the switches are not amazing. Sadly they are not mine to replace - they are a clients (I never set this up!).

 

I think Jumbo Frames makes a difference I'm just debating turning off Flow control to see if the spikes stop - From initial investigations they are only hanging when Veeam is removing its snapshots but as they do hourly replication to a DR sites its a larger issue then once an evening.

 

On a normal Paused frame what is the time period they are held for?

[SYNACK]

Any particular reason for the jumbo frames? I never found it made much difference with 1Gb iSCSI. The 1810 really is a low end switch, even if it is all Gb and fanless.

 

Jumbo frames cuts down on the packaging and increases the room for 'product' in the data stream and buffers however DmcCoy says the 1810 is a rather low end switch with small buffers which could be contributing to the issue. Disabling jumbo frames may crank the whole system down in performance but may level out the performance.

[glennda]

Jumbo frames cuts down on the packaging and increases the room for 'product' in the data stream and buffers however DmcCoy says the 1810 is a rather low end switch with small buffers which could be contributing to the issue. Disabling jumbo frames may crank the whole system down in performance but may level out the performance.

 

At the moment I would rather keep it on and only disable it as a last resort due to running VDI pools etc. I'm going to try disabling Flow Control first to see if that lowers the overheads on the switching - will have to wait as i'm not brave enough to do it during the day although they are supposedly in a fault tolerant cluster!

[SYNACK]

At the moment I would rather keep it on and only disable it as a last resort due to running VDI pools etc. I'm going to try disabling Flow Control first to see if that lowers the overheads on the switching - will have to wait as i'm not brave enough to do it during the day although they are supposedly in a fault tolerant cluster!

 

With the flow control turned off it may get worse as the bottleneck won't be corrected as quickly or as well but it is worth a shot.

[glennda]

With the flow control turned off it may get worse as the bottleneck won't be corrected as quickly or as well but it is worth a shot.

 

If it causes significate problems I can turn it back on again easily where as I would need to restart the Networking on the VM hosts to disable Jumbo Frames which means taking down the servers in my evening and I would rather go and see the Dark Knight this evening

[SYNACK]

If it causes significate problems I can turn it back on again easily where as I would need to restart the Networking on the VM hosts to disable Jumbo Frames which means taking down the servers in my evening and I would rather go and see the Dark Knight this evening

 

It is a great movie, saw it at midnight on release day so we got free V energy drink and stuff as part of their promo as well.

[glennda]

It is a great movie, saw it at midnight on release day so we got free V energy drink and stuff as part of their promo as well.

 

lucky you where not in america then! I'm hoping to watch in imax but depends if there is availability for it

[SYNACK]

lucky you where not in america then! I'm hoping to watch in imax but depends if there is availability for it

 

Yea, saw it on the 'Megascreen' here, the iCantBeleiveItsNotIMAX sized screen. Heard about the american incident the next night when it happened as a breaking news ticker and was glad that we got free V instead.

[DMcCoy]

 

I know what they are, but I've spent *a lot* of my time since ESX 2.5 doing various real and I/O meter tests,

 

CPU usage *will* be better, but I've never found it much better, and it was also slower at times depending on what I was testing with, never enough to worry about the effort to make it work.

 

Although it's much easier to configure now I must admit. I had the same issue with the Broadcom iscsi offload for the 5709s, got it all working but hardly seemed worth the effort too!

Edited July 23, 2012 by DMcCoy

[SYNACK]

I know what they are, but I've spent *a lot* of my time since ESX 2.5 doing various real and I/O meter tests,

 

CPU usage *will* be better, but I've never found it much better, and it was also slower at times depending on what I was testing with, never enough to worry about the effort to make it work.

 

Although it's much easier to configure now I must admit. I had the same issue with the Broadcom iscsi offload for the 5709s, got it all working but hardly seemed worth the effort too!

 

Yea in Hyper-V it is a whole two clicks to switch it on and then just enabling it on the switch. It was that easy that it seemed rude not to and it did speed things up a little in throughput terms with our setup.