Xenserver 5.5 w/ Sun 7110 NFS SR Slow writes in guest VM's

[buccy]

Hi All,

 

I know that several people on here must have a similar setup and I am wondering if they experience the same behaviour or have a fix/workaround. Here is the setup/issue:

 

3 Xenserver 5.5 hosts in a pool with a NFS SR hosted on the Sun 7110, when benchmarking within a guest VM Read speeds are great around 80 MB/s but write speeds are just horrible I'm lucky it I get over 10 MB/s! (see screenshot) There are numerous threads over on the Citrix forums regarding this problem but none seem to offer a solution. Is there someone out there with a similar setup that is able to test the write performace in a guest VM and post the results? Does anybody have an idea how to fix this issue? Cheers.

[Ric_]

What are you using to do your performance test? I'll happily check my setup but would prefer to use the same tool to give an accurate comparison.

[buccy]

Its just a quick and dirty benchmark program called crystal disk mark, but I have seen the same results using ATTO disk benchmark as well. If you could post your results that would be great.

[Ric_]

@buccy: You got a link?

[buccy]

Yeah sorry here Crystal Dew World

[webman]

We have two storage pools on our 7110 - one mirrored and one double-parity raid. We're also using NFSv3 to (currently) a single XenServer 5.5 host.

 

Mirrored volume: Read: 99.96 MB/s. Write: 24.65 MB/s.

 

Double-parity volume: Read: 94.19 MB/s. Write: 19.82 MB/s.

 

I don't know if that's "standard" for our configuration or not. If Ric_ comes back with figures twice our write speed I think ours might need investigating...

 

As a comparison, my workstation SATA disk benchmarks at 60.94 MB/s read and 11.62 MB/s write.

Edited June 11, 2010 by webman
Added XenServer version.

[buccy]

Your local results sound a bit off, my workstation gets 85.2 MB/s read and 76.82 MB/s write, I get similar results to these on a VM running on local storage. Thanks for posting your results.

[irsprint84]

5.6 is out, tried the upgrade? even the 5.5 sr2 (I think!) fixed alot of problems

[buccy]

I am a bit weary of upgrading to 5.6 as a number of issue's with it have been raised. I am going to setup a Openfiler box and test the performance from that, in the mean time is there anyone else who experiences slow writes in Xenserver?

[buccy]

I have done some further testing and enabled flow control on the SAN switch, this has made performance slightly worse!

[cookie_monster]

I have done some further testing and enabled flow control on the SAN switch, this has made performance slightly worse!

 

 

Really? I remember reading somewhere that flow control is considered essential for this kind of setup. I don't remember a performance hit when we enabled it I'll have to check.

[buccy]

Cookie any chance you could post some benchmark results from either Crystal Disk Mark or ATTO Disk benchmark, it would be reassuring to know if what I am experiencing is common so I can stop pulling my hair out! Edited June 15, 2010 by buccy

[cookie_monster]

Cookie any chance you could post some benchmark results from either Crystal Disk Mark or ATTO Disk benchmark, it would be reassuring to know if what I am experiencing is common so I can stop pulling my hair out!

 

 

How was you testing? Were you just uploading data from a client to a Windows VM? I'll give it a try when I get a sec but we're using iSCSI here so it might not help.

[buccy]

iSCSI numbers would be useful also as I have begun investagating using it if I can't improve the NFS numbers. Both programs I listed just create temp files on the guest VM and are removed after the run. ATTO is lightweight and no install neccessary, so might be the best option.

[cookie_monster]

I wonder how the SAN is affected when it's busy. I just got 94.44MB/s Read and 5MB/s Write but it was busy at the time. We are using the old iSCSI stack though, I'm planning a move to the latest release in the summer. Edited June 16, 2010 by cookie_monster

[buccy]

cookie what type of VM's do you use? Generally the slow writes is not a real problem but on a couple of our APP servers (SQL databases) it is starting to cause problems, the main one is that it is so much slower than a physical box. I have started testing iSCSI SR's and while the read speeds are generally slower than NFS the writes are at least twice as fast, untill like you mentioned, during busy periods speeds basically halve.

 

Ric_ have you had a chance to test your system?

[cookie_monster]

I'm using a mix of Windows 2003 and 2008 servers hosted on Xenserver 5.5 (if that's what you're asking). Apparently the latest Xenserver release fixes some performance issue I need to have a look through the release notes.

[buccy]

I was wondering what kind of servers, do you have any SQL or Exchange VM's? I have updated my test box to 5.6 and tested the NFS performace and there is no change unfortunately, iSCSI performace is marginally better.

[Ric_]

Sorry for the delay in posting my results.

 

I have benchmarkerd several VMs.

 

1. XenServer 5.5 running a 2003 x64 server (the only VM on that host's local storage)

2. XenServer 5.5 running a 2008R2 server (the only VM on that host's local storage)

3. XenServer 5.6 running a 2008R2 server (the only VM on that host's local storage)

4./5. XenServer 5.6 running 2 Server 2008R2 VMs on local storage

 

I have only benchmarked the VMs I run on local storage so that I can get a fair comparison. All VMs have the relevant version of the XenServer Tools package applied. The XenServer 5.5 hosts have 2.5" 15k rpm SAS disks and the XenServer 5.6 hosts have 3.5" 15k rpm SAS disks. The 5.6 boxes also potentially had load at the time of testing.

[buccy]

Thanks Ric_ could you possible do one more test on a VM running on shared storage NFS or iSCSI? Cheers.

[webman]

Couple of questions Ric_: How are your Xen hosts connected to the 7110 (NIC configuration - bonded/teamed?) NFS or iSCSI? What type of volume(s) on the 7110?

 

Thanks in advance

[Ric_]

Thanks Ric_ could you possible do one more test on a VM running on shared storage NFS or iSCSI? Cheers.

 

Sure... I have no comparison between XenServer 5.5 and XenServer 5.6 though as the hosts using shared storage are both still on 5.5.

 

I have attached benchmarks from a 2008R2 server and a 2003 x64.

 

The shared storage is off a Sun Storage 7110 connected via a pair of gigabit ethernet connections in LACP. Both VMs were on the same host and under load at the time. There were 17 active VMs running from shared storage at the time of testing.

[Ric_]

Couple of questions Ric_: How are your Xen hosts connected to the 7110 (NIC configuration - bonded/teamed?) NFS or iSCSI? What type of volume(s) on the 7110?

 

Thanks in advance

 

Cheeky with 'Thanks in advance'!!!

 

pair of gigabit copper connections to HP 2610G-24 and then a pair of copper links to each host. I have just 1 volume (a limitation at the time of setup) and it's set up as double parity RAID. As mentioned, I use NFS. I'm also using CIFS to directly serve files off the boxes too.

 

Also see http://www.edugeek.net/forums/hardware/35639-sun-storage-7110-performance.html

[webman]

Thanks Ric_

 

I did read the 7110 Performance thread before we set ours up (probably a dozen times or more); but the varying results seemed inconclusive. It seems the three variables of network, storage protocol and filesystem all have an impact on what you'll actually get with XenServer and Sun storage. As always, 'YMMV'.