Xenserver 5.5 w/ Sun 7110 NFS SR Slow writes in guest VM's

[buccy]

Hi All,

I know that several people on here must have a similar setup and I am wondering if they experience the same behaviour or have a fix/workaround. Here is the setup/issue:

3 Xenserver 5.5 hosts in a pool with a NFS SR hosted on the Sun 7110, when benchmarking within a guest VM Read speeds are great around 80 MB/s but write speeds are just horrible I'm lucky it I get over 10 MB/s! (see screenshot) There are numerous threads over on the Citrix forums regarding this problem but none seem to offer a solution. Is there someone out there with a similar setup that is able to test the write performace in a guest VM and post the results? Does anybody have an idea how to fix this issue? Cheers.

[Ric_]

What are you using to do your performance test? I'll happily check my setup but would prefer to use the same tool to give an accurate comparison.

[buccy]

Its just a quick and dirty benchmark program called crystal disk mark, but I have seen the same results using ATTO disk benchmark as well. If you could post your results that would be great.

[Ric_]

@buccy: You got a link?

[buccy]

Yeah sorry here Crystal Dew World

[webman]

We have two storage pools on our 7110 - one mirrored and one double-parity raid. We're also using NFSv3 to (currently) a single XenServer 5.5 host.

Mirrored volume: Read: 99.96 MB/s. Write: 24.65 MB/s.

Double-parity volume: Read: 94.19 MB/s. Write: 19.82 MB/s.

I don't know if that's "standard" for our configuration or not. If Ric_ comes back with figures twice our write speed I think ours might need investigating...

As a comparison, my workstation SATA disk benchmarks at 60.94 MB/s read and 11.62 MB/s write.

[buccy]

Your local results sound a bit off, my workstation gets 85.2 MB/s read and 76.82 MB/s write, I get similar results to these on a VM running on local storage. Thanks for posting your results.

[irsprint84]

5.6 is out, tried the upgrade? even the 5.5 sr2 (I think!) fixed alot of problems

[buccy]

I am a bit weary of upgrading to 5.6 as a number of issue's with it have been raised. I am going to setup a Openfiler box and test the performance from that, in the mean time is there anyone else who experiences slow writes in Xenserver?

[buccy]

I have done some further testing and enabled flow control on the SAN switch, this has made performance slightly worse!

[cookie_monster]

I have done some further testing and enabled flow control on the SAN switch, this has made performance slightly worse!

Really? I remember reading somewhere that flow control is considered essential for this kind of setup. I don't remember a performance hit when we enabled it I'll have to check.

[buccy]

Cookie any chance you could post some benchmark results from either Crystal Disk Mark or ATTO Disk benchmark, it would be reassuring to know if what I am experiencing is common so I can stop pulling my hair out!

[cookie_monster]

Cookie any chance you could post some benchmark results from either Crystal Disk Mark or ATTO Disk benchmark, it would be reassuring to know if what I am experiencing is common so I can stop pulling my hair out!

How was you testing? Were you just uploading data from a client to a Windows VM? I'll give it a try when I get a sec but we're using iSCSI here so it might not help.

[buccy]

iSCSI numbers would be useful also as I have begun investagating using it if I can't improve the NFS numbers. Both programs I listed just create temp files on the guest VM and are removed after the run. ATTO is lightweight and no install neccessary, so might be the best option.

[cookie_monster]

I wonder how the SAN is affected when it's busy. I just got 94.44MB/s Read and 5MB/s Write but it was busy at the time. We are using the old iSCSI stack though, I'm planning a move to the latest release in the summer.