Sun Storage 7110 Performance

[Ric_]

As promised, I did a little testing. The following results are for Windows Server 2003 x64 running under XenServer 5.0 Update 3. The VM was given 2GB RAM and 2 virtual CPUs.

 

The 7110 is connected to the XenServer hosts as shown in the image - bonded gigabit copper links via a HP Procurve 2610G-24 switch. The XenServer hosts are told to talk to the storage on a special subnet which is VLANed off from the network too. Other VMs were running during the testing.

 

I used IOMeter with settings taken from recommendations at Operating Systems and Benchmarks - Part 5 (I have attached the config file too).

 

The full results are enclosed within the attached archive. A summary follows:

 

iSCSI:

* 72.92 average IOps

* 0.81 average MBps

* 16ms average response time

 

NFS:

* 162.42 average IOps

* 1.72 average MBps

* 7.71ms average response time

 

I also incluse a comparison with my workstation (a HP xw6200 with a SATA hard disk) to show some context:

* 107.34 IOps

* 1.18 MBps

* 9.31ms response time

 

These are very basic, simulated tests but do suggest NFS to be the quicker connection type under these circumstances.

 

I intend to run a longer test when time permits to give a better idea of speed. I will also add a comparison between file transfer rates straight to the Sun Storage 7110 via CIFs and to the storage via a VM running Server 2003.

iometer_results.7z

[Duke]

Nice one, thanks a lot! It's really good to see some real-world figures, although I must say I'm surprised there was that much of an improvement with NFS when most people just assume iSCSI is the way to go for block-level storage. Looks like (for now) NFS is the best bet for virtualisation, I'll have to have a play with ours...

 

Cheers,

Chris

[linescanner]

The other advantage of NFS is that you will get more info from the Diagnostic tools on the 7000 series.

 

Big thanks Ric for publishing this. It backs up what we expected.

[Butuz]

Very good rick!

 

Hmm - now to find out exactly what NFS is and what is does! (noob)

 

Butuz

[linescanner]

Very good rick!

 

Hmm - now to find out exactly what NFS is and what is does! (noob)

 

Butuz

 

Ahh !! the innocence of the Windows Admin

 

Andrew, give me a bell tomorrow and I can point you in the right direction.

[DMcCoy]

I know this is of no use for testing 7110s but it can help give the numbers some context

 

Things we still need to know Ric_! RAW/Formatted drive, number of drives in the RAID group, RAID type for the RAID group, drive speed.

 

Some numbers using the test supplied for my own SANs. I made a change to keep the iobw.tst size to 2GB (need to keep it largeish to avoid cache throwning the numbers, but also want to not fill the drive as I'm using it on existing partitions!).

 

 

HP2012i, iscsi from ESX host (test within VM). 14 x 15k RPM SAS drives in RAID 6 group

* 164.8 average IOps

* 1.70 average MBps

* 6.09ms average response time

 

EMC CX300 Fibre channel (test within VM). 10 x 10k RPM FC drives in RAID 5 group

* 234.92 average IOps

* 2.54 average MBps

* 4.25ms average response time

 

 

 

It's worth remembering that the CX300 is from 2005, and the HP is 2008. Although the CX was almost 3 times the price of the later HP. iscsi has caught up a long way, but the queuing and throttling system for fibre channel is more efficient than iscsi.

[Duke]

Ahh !! the innocence of the Windows Admin

 

Andrew, give me a bell tomorrow and I can point you in the right direction.

 

*cough*

 

Er, +1 on that.

 

Has anyone got any good NFS docs pages, particularly with regards to security? iSCSI has initiator security and CHAP, what does NFS provide?

 

Cheers,

Chris

[Butuz]

Ric am I being thick? Where is the attatched config file?

 

Cheers

 

Butuz

[Ric_]

Ric am I being thick? Where is the attatched config file?

 

Unless I'm being thick, it's in the 7-zip file which I almost certainly attached.

 

Things we still need to know Ric_! RAW/Formatted drive, number of drives in the RAID group, RAID type for the RAID group, drive speed.

 

I knew I would forget some vital piece of information.

 

My 7110 is configured thus:

* 2x 146GB 10,000rpm SAS disks mirrored for storage system OS

* 14x 146GB 10,000rpm SAS disks configured as double parity RAID (1 of which is a spare) - gives 1.4TB usable space.

[Guest Guest]

Ric_

Looking at your summary you have included the anomylous results in your averages; you shouldnt do this. The 4th set of iSCSI results should be binned, and an average taken from the other 3.

 

Also your NFS results are massively varied. You should really do another half dozen tests, take out the anomylous results, and re-average.

 

 

Even taking that into account i believe something isnt working properly for you. My peliminary iSCSI results are comparable to DMcCoys, ie much better than what you are getting, and our kit is half the spec of yours!!!!

Edited May 21, 2009 by Guest

[Soulfish]

I'm hoping tomorrow I may be able to spend a couple of hours testing our 7110 so I'll post up my results as soon as I get them

[Guest Guest]

My results are as follows;

 

Xenserver 5 update 3, iSCSI SR to 1 Openfiler san. Crappy Dell gigabit switch. Single 1gb nic link between the san and xen in the same arrangement as Ric but without the vLan.

 

Openfiler is installed on a Dell poweredge 2950 with a xeon 5420, 2gb ram, 6x 750gb 7.2k SAS in raid 5.

 

(We've actually got a few nics to go in, and when in production the openfiler will have HA by way of a DRBD mirroring... but thats of no consiquence for these results)

 

Server 2003 32bit with 1 vCPU & 512mb ram was used as the testing VM. iSCSI was attached using MS iSCSI initiator. Results are as follows;

 

* 152.12 average IOps

* 1.64 average MBps

* 6.6ms average response time

 

 

 

For referance my laptop with a 5200prm sata hard drive;

 

* 55.65 average IOps

* 0.57 average MBps

* 17.9ms average response time

 

 

As you can see my laptop is comparable to Ric_s iSCSI results, and my iSCSI results are miles ahead even with half the spec (both VM and SAN) that Ric does, hense my opinion that something isnt quite right in his set up.

Edited May 21, 2009 by Guest

[Butuz]

Once my Fire X4150s and 7110s come I will do some performance testing on ESXi / XEN and iSCSI / NFS. See how it stacks up.

 

Butuz

[Ric_]

@j17sparky: I deliberately left in what you call anomylous results because it shows that there is a wide variation. The tests werte performed on a live setup which was in use at the time by other VMs.

 

The iSCSI results are poor... Sun know that they are poor and intend to improve iSCSI performance.

 

As my OP states, I will be running a 'better' test when I have time... I might do a couple of 24 hour tests this weekend and include database simulations as well.

[Duke]

I'm reviving this thread as I'm just doing my own performance testing and I think I need a bit of help...

 

Ric_ - I've downloaded your config file and that's what I'm using with IOMeter. I have a Windows XP SP2 virtual machine set up, stored on our 7410. I've started up IOMeter, imported the config then basically just hit Start, have I missed something? The VMs are 1 vCPU, 512MB RAM and 12GB HDD. IOMeter fills up the HDD then does several minutes of testing.

 

The reason I ask if I've done something wrong is that my results are very different to everyone else. I assume I should just be reading the figures straight off the Results Display tab?

 

Sun 7410

Total IOPS: 466.30

Total MBps: 4.95

Average Response: 2.14ms

 

I must be missing something, but from what I can see the disk usage graphs in vCenter certainly tie up with the 4.95 MBps. Any suggestions anyone?

 

EDIT: I'm running the same tests on a NetApp FAS2020 at the moment and while there's still 10mins left on the test it's giving me figures similar to the rest of you (77 IOPS, 0.81MBps). Can anyone explain my Sun 7410 results, especially considering this is running over a 100Mb network right now?

 

EDIT 2: Final results for the FAS2020 are in now and seem fairly similar to everyone else:

 

NetApp FAS2020

Total IOPS: 75.89

Total MBps: 0.87

Average Response: 13.17ms

 

 

Many thanks in advance,

Chris

Edited June 10, 2009 by Duke

[Butuz]

Duke - I think your on the right track, what you did with RICs config file is exactly what I have done (download IOMETER, install, open RICs file, point at the correct hard drive, run and wait 10 mins).

 

I think your investment in the 7410 shows in your performance stats. What's your exact 7410 config?

 

My Results are as follows.

 

Server Stats:

Sun Fire X4150 running VMWare ESXi 3.5 Update 4

8 x 3.16Ghz Xeon Cores

16GB RAM

4 x 146GB SAS drives in Raid 10

 

Network Stats:

HP ProCurve 2810-48G Switch

4 x 1GB Trunk to Fire X4150

4 x 1GB Trunk to Storage 7110

SAN data NOT vlanned off currently (i.e set up on curriculum network)

 

Test VM Stats:

Windows 2003 Std 32 Bit.

2GB RAM Allocated

 

Test1 - Control Test using ESXi local Storage:

Test uses local Fire 4150 4 x 146GB Internal Hard Drives in Raid 10

Total IOPS: 223.06

Total MBps: 2.31

Average Response: 4.48ms

CPU Utilisation: 1.97%

 

Test2 - NFS 7110 Test using Raid 0:

Test uses remote 7110 14 x 146GB drive Raid0 Filesystem over NFS

Total IOPS: 622.61

Total MBps: 6.93

Average Response: 1.60ms

CPU Utilisation: 1.13%

 

Test2 - NFS 7110 Test using Raid 6:

Test uses remote 7110 14 x 146GB drive Raid6 Filesystem over NFS

Total IOPS: 356.83

Total MBps: 3.84

Average Response: 2.80ms

CPU Utilisation: 1.07%

 

As you can see - very impressive results. Even the internal storage of the Fire X4150s is damned fast. This Sun stuff is pie hot!

 

Butuz

Edited June 10, 2009 by Butuz

[Duke]

Duke - I think your on the right track, what you did with RICs config file is exactly what I have done (download IOMETER, install, open RICs file, point at the correct hard drive, run and wait 10 mins).

 

I think your investment in the 7410 shows in your performance stats. What's your exact 7410 config?

 

Does look good so far doesn't it? I'm going to run the figures again later today if I get time and I'll also test NFS. If I can find some room I'll put the ESX hosts in the server room on the same Gb switch as the SAN, at the moment it's over 100Mb as they're in my office which is about five switches away from the server room...

 

Sun 7410 config is as follows:

 

7410 Controller

16GB RAM

2x 2.3 Quad-Core CPUs

1x 100GB Readzilla

 

J4400 SAS Array

22x 1TB 7200 SATA

2x 18GB Logzilla

 

Clearly the flash accelerators are playing a big part here, but to see 5x-6x the performance of a NetApp box at about half the price... wow!

 

Chris

[Butuz]

Clearly the flash accelerators are playing a big part here, but to see 5x-6x the performance of a NetApp box at about half the price... wow!

 

Chris

 

Yes the NetApp box results are truly shocking!!! Was it being used / under load when you ran the tests???

 

It's a shame I don't have HP, DELL, Hitatchi SANs lying around - it would be VERY interesting to do a performance comparison.

 

Butuz

[Duke]

Nope, it's mainly used as a D2D backup and archiving box so does nothing during the day. I shouldn't really jump to any conclusions, this is the first time I've ever had reason to benchmark the NetApp filer so there may be a configuration issue that's my fault.

 

I'd love to see how these figures compare to a server running SANMelody as that's a solution that was strongly suggested to us. I think SANMelody looks like a great product and obviously works out a lot cheaper than this mid-end SAN hardware, but can it come anywhere close to these kinds of figures?

 

If we were to standardise a full benchmarking system, say Ric's config, a specific ESX and virtual machine setup, etc, would a few other people be willing to do some tests and put their figures forward?

 

Cheers,

Chris

[Duke]

I just got 580 IOPS with SSD write caching on the 7410 enabled, although I have run into a bit of a bug on the Sun box doing so (nothing major, just a BUI glitch under heavy load). My question is can anyone explain how to setup NFS with ESX on the Sun 7000? I'm not particularly familiar with NFS and my last attempt ended up with THIS.

 

Many thanks in advance to anyone who can explain or point me towards a guide or how-to!

Chris

[Butuz]

Well guys I managed to finally get round to doing some NFS v iSCSI performance testing on my 7110s as I was interested that people had noted poor iSCSI performance and wanted to see if I could replicate this. Both my 7110s have been upgraded to the latest firmware (2009-04-10-3-0-1-1-16).

 

All performance testing was done with the following networking conditions:

Test VM - Windows 2003 Server with 4 x 3.16Ghz vCPU and 4GB vRAM

SANs - both of them only had a single 1GB link configured (no trunking etc)

ESX host was a SUn X4150 and had a single 1GB link configured to the SAN and local storage was 2 x 146GB SAS drives in Raid 1

No live VM's running - i.e purely a test spec setup.

 

Control Test: Local Storage configured as RAID1

Total I/O Per Second: 208.45

Total MBs per Second: 2.14

Average Response: 4.7968

Maximum Response: 114.6618

CPU Utilisation: 0.74%

 

SAN-001 - Configured as Double Parity Raid (Raid 6). 1.4TB Usable

 

SAN-001 NFS v3:

Total I/O Per Second: 348.52

Total MBs per Second: 3.51

Average Response: 2.8685

Maximum Response: 1750.4661

CPU Utilisation: 0.89%

 

SAN-001 iSCSI:

Total I/O Per Second: 173.92

Total MBs per Second: 1.90

Average Response: 5.7491

Maximum Response: 2734.5700

CPU Utilisation: 0.66%

 

SAN-002 - Configured as Mirrored (Raid 10?) 0.8TB Usable

 

SAN-002 NFS v3:

Total I/O Per Second: 508.82

Total MBs per Second: 5.73

Average Response: 1.9668

Maximum Response: 1198.8586

CPU Utilisation: 1.33%

 

SAN-002 iSCSI:

Total I/O Per Second: 201.25

Total MBs per Second: 2.26

Average Response: 5.0480

Maximum Response: 260.3086

CPU Utilisation: 0.66%

 

As you can see iSCSI performance on the 7110 is shocking. I will be doing these test again soon with 4x1GB trunked links to see if that helps iSCSI perfromance along. I have my doubts!

 

Will deffo be using NFS for my ESXi Host's VM storage.

 

Butuz

Edited July 13, 2009 by Butuz

[Duke]

Looks good on NFS.

 

Still need to get mine set up, I'll try to have a look at that today.

 

Good news on de-dupe everyone: Sun setting dedupe up for ZFS ? The Register

 

Chris

[john]

Sounding Good Very much a good selling point to Education customers that one

[Ric_]

@Butuz: NFS better than local storage... now that's good to see!

 

Backs up what I found on older firmware with iSCSI poorer than NFS too

 

So as long as you don't use HyperV (which doesn't officially support NFS as a storage medium), it looks like NFS is the way to go still

[Butuz]

Yep I am really pleased with the performance results on NFS. My VMs should be happy!

 

I am using ESXi 4 too which apparently has quite a bit of iSCSI improvement compared to ESXi 3.5.

 

Will deffo do some quick tests with trunking to see if it makes any difference to the results.

 

Butuz