Ric_ Posted July 15, 2009 Author Posted July 15, 2009 I am using ESXi 4 too which apparently has quite a bit of iSCSI improvement compared to ESXi 3.5. Only improves iSCSI performance if it doesn't suck to begin with on your storage
Butuz Posted July 15, 2009 Posted July 15, 2009 Indeed Now where did that chap from Sun go to? Why is iSCSI performance so poor? Butuz
Ric_ Posted July 15, 2009 Author Posted July 15, 2009 Now where did that chap from Sun go to? Why is iSCSI performance so poor? Sun are new to iSCSI but they've done NFS since the year dot... hence the massive performance difference. I believe that the Solaris iSCSI initiator is getting some work done to it and it will be better in the future.
pete Posted July 15, 2009 Posted July 15, 2009 Has anyone compared vanilla OpenSolaris iSCSI vs NFS performance? Just wondered if the problem was with Solaris or the implementation on the box.
Duke Posted July 15, 2009 Posted July 15, 2009 Hey all, NFS has decided to work for me so I've got some figures too. I tried to set up NFS with ESX4 the way I normally do and it just worked straight away with no errors for once. The only change I’ve made is to upgrade the firmware on the 7410 so maybe that fixed something? There was some background traffic on the 7410 as it’s in production use, but very little and only for a very short time during the tests. VMware vSphere with ESX 4.0.0 Hosts are Dual Core 3.0GHZ with 3GB RAM Guests are XP SP2 with 1 vCPU, 512 MB RAM, 12GB disks (VMware Tools installed) Network connection is 100MB iSCSI Testing to Sun 7410 Total IOPS: 188.48 Total MBps: 2.02 Avg I/O Response: 5.3 Max I/O Response: 5007.2 CPU %: 2.2 No idea why this is such a poor result, and far lower than I had previously. Caching is enabled on the LUN so it should be flying, same as before. Maybe the newer firmware has made things worse for people using iSCSI with flash accelerators? While Iometer was doing the disk preparation the IOPS measured by VSphere were way better than when the test was actually running. NFS Testing to Sun 7410 Total IOPS: 431.22 Total MBps: 4.61 Avg I/O Response: 2.3 Max I/O Response: 1011.8 CPU %: 5.4 Anyone know why you can't see disk activity/performance in vSphere when you use NFS? The graphs work fine with iSCSI but there's no option to view the disk data on a guest that uses NFS. Quick question: When I add an iSCSI datastore to one of my ESX hosts it automatically appears on all my hosts. With NFS I had to add it to them both manually. Is this correct and if so what’s the reasoning behind it? Cheers, Chris
Duke Posted July 21, 2009 Posted July 21, 2009 Are you using NFSv3 or NFSv4? I'm pretty sure it was NFSv3, which is what ESX appeared to default to. Not sure if there's an option to tell ESX4 to use NFSv4, or if it even supports it? Cheers, Chris
Ric_ Posted July 21, 2009 Author Posted July 21, 2009 I used NFS v3... I didn't see an option for v4 so I didn't tell it to use it - I know it's v3 from the pretty graphs
nickelj Posted March 15, 2010 Posted March 15, 2010 (edited) Hi Everyone, Since you shared with me and it helped me out, I thought I would share back. Our environment is VMware vSphere (ESX 4) with HP GB switches to a single Sun 7110. The servers are Sun X4150's with 32 GB RAM and 2.83 GHz Xeon CPU's. I have vlanned off the storage onto private network, but I have not teamed anything for this test. All tests were done with a single NIC from the servers and a single NIC on the 7110. The version of firmware on the 7110 was 2010.02.09.0.0,1-1.9 For IOMeter, I used the same icf file listed at the front of this thread. I tested from within a Windows 2003 VM and I didn't format the partition in Windows 2003. There were no other VM's accessing the Sun 7110 at the time of the tests. Also...some information about how the Sun 7110 is setup for the tests: For NFS - Data Dedupe is off, Data Compression is off, Synchronous Write Bias is set to Throughput and Database Record size is 128K For iSCSI - Data Dedupe is off, Data Compression is off, Synchronous Write Bias is set to Throughput, Block sizes set to 64K and turned Write Cache on. Obviously if you turn things like Data Dedupe on...it is gonna slow things down a bit. I thought a comparison of NFS and iSCSI would be good, plus a comparison of Double Parity Raid to RAID 10 (mirrored). You know that mirrored should be faster, but how much faster...will it be worth it to you? This may help you decide. So....for Double Parity Raid- NFS: Total I/O's per Second: 323.13 Total MB's per Second: 3.44 Average I/O Response Time: 3.0941 Maximum I/O Response Time: 330.4260 % CPU Utilization: 1.18 iSCSI: Total I/O's per Second: 2648.69 Total MB's per Second: 28.61 Average I/O Response Time: 0.377 Maximum I/O Response Time: 475.4195 % CPU Utilization: 5.51 Mirrored (RAID10)- NFS: Total I/O's per Second: 557.49 Total MB's per Second: 5.77 Average I/O Response Time: 1.7932 Maximum I/O Response Time: 513.4279 % CPU Utilization: 1.58 iSCSI: Total I/O's per Second: 2211.62 Total MB's per Second: 24.31 Average I/O Response Time: 0.4516 Maximum I/O Response Time: 1719.4337 % CPU Utilization: 4.68 Obviously the write cache is having a significant effect. So...I did one last test with iSCSI with the Write Cache disabled. Also..interestingly, the RAID10 iSCSI performance was not higher with Write Cache enabled. Notice that the response time with iSCSI with Write Cache enabled is much faster than NFS. The other problem I have with the Write Cache is I am not convinced that the above numbers would be accomplished during "real world" usage - especially after the cache is full. Here are the results of iSCSI with Write Cache disabled. iSCSI RAID10 - Write Cache Disabled Total I/O's per Second: 638.14 Total MB's per Second: 6.82 Average I/O Response Time: 1.5665 Maximum I/O Response Time: 215.4945 % CPU Utilization: 2.04 iSCSI Double Parity Raid - Write Cache Disabled Total I/O's per Second: 120.50 Total MB's per Second: 1.29 Average I/O Response Time: 8.2981 Maximum I/O Response Time: 1001.3984 % CPU Utilization: 1.03 So...without the cache....RAID10 is still faster, but Double Parity Raid looks bad with iSCSI compared to NFS. So...who is the winner would depend on what you need - if you need space and go with RAID6 (Double Parity Raid), then NFS is the winner. If you want the best performance and can sacrifice space...then iSCSI with RAID10 is better. Jim Edited March 15, 2010 by nickelj 1
Duke Posted March 15, 2010 Posted March 15, 2010 Looks good, thanks for doing a variety of tests and RAID types!
Guest Guest Posted March 15, 2010 Posted March 15, 2010 (edited) How come nickeljs iSCSI performance is so high? I was under the impression NFS was better on the SUN boxes. Any ideas? //Or is it that iSCSI has better max IO when benchmarking, but NFS is better in real life with multiple concurrent connections? Edited March 15, 2010 by Guest
Duke Posted March 15, 2010 Posted March 15, 2010 Must admit I was a bit surprised by that too, I know Sun improved iSCSI a lot with the COMSTAR stack, but I thought NFS was still better for VMware. These were my Q2.5 figures over 100Mb network: Q2.5 NFS IOPS: 596.42 MBPS: 6.34 Avg Response: 1.6761 Max Response: 255.5736 CPU Util: 7.02 iSCSI IOPS: 254.76 MBPS: 2.53 Avg Response: 3.9247 Max Response: 4177.26 CPU Util: 3.19 Never got around to comparing iSCSI after I'd upgraded to Q3. Chris
cookie_monster Posted March 15, 2010 Posted March 15, 2010 I'm still on the old iSCSI stack at the moment and it's adequate, i've been told by Sun that it's a big improvement in the new update.
Butuz Posted March 15, 2010 Posted March 15, 2010 Hmm I think I'm gonna have to reconfigure my 7110's to Raid 10 and investigate iscsi write cache! Those new numbers look really good! Butuz
nickelj Posted March 15, 2010 Posted March 15, 2010 Just in case this wasn't clear...this was using 1 gigabit NICs. In my experience...iSCSI usually does outperform NFS in "real world" situations with other devices. The Comstar code in the new iSCSI for the Sun 7000 series of storage units makes a big difference. However, as indicated - NFS is better if you are going to go with RAID6 - double parity raid. I wouldn't trust the Write Cache enabled numbers for iSCSI - the cache will fill up pretty fast and is not big enough on a 7110 (it is only 3-4 GB) to sustain the throughput. I did some testing under load and the results were very "all-over-the-place" with iSCSI and Double Parity Raid. Much more consistent with NFS and Double Parity Raid. However, the RAID10 (mirroring) with iSCSI was really good - the minimum figures would be the ones I posted above without Write Cache Enabled and they would only get better with it enabled. The units that have the SSD's (everything except the 7110 I think) should not suffer the same performance hit with Double Parity Raid and iSCSI. When I moved from iSCSI and Double Parity Raid this weekend to a Mirrored (RAID10) with iSCSI, the performance difference was huge - users noticed right away and my ATTO benchmarks from within the VM's hit over a 100 mbps and were consistently good. I would highly recommend upgrading to the version I listed in my post. Oh and by the way...it now has a Dedupe checkbox! Jim 2
SLMHC Posted April 28, 2010 Posted April 28, 2010 (edited) I just upgraded to 2010.02.09.0.2,1-1.13 tonight from the 2009.09.01.4.1,1-1.13 and am seeing all the nice new features. Since I have this box in production I imagine that I would have to back up all the existing VMs, wipe the double parity raid config and reconfig using RAID 10 to get the huge speed increases iSCSI gives us with the new stack? EDIT: Wooohooo! AD join finally worked with this build! Sorry aboot that, going back to being a reserved Canadian now... Edited April 28, 2010 by SLMHC Rocking 2010 Q1 Update!
nickelj Posted April 28, 2010 Posted April 28, 2010 Yes...unfortunately there is no way to change raid types without wiping the data. But based on my experience...it is well worth it. Even if you do leave it at Double Parity Raid, the unit performs much better than it previously did, so just upgrading is worth it. Jim
pete Posted April 28, 2010 Posted April 28, 2010 I just upgraded to 2010.02.09.0.2,1-1.13 tonight from the 2009.09.01.4.1,1-1.13 and am seeing all the nice new features. Since I have this box in production I imagine that I would have to back up all the existing VMs, wipe the double parity raid config and reconfig using RAID 10 to get the huge speed increases iSCSI gives us with the new stack? EDIT: Wooohooo! AD join finally worked with this build! Sorry aboot that, going back to being a reserved Canadian now... *suspects SLMHC may be a merkin in disguise* (loads of US backpackers - IME - pretend to be Canadian, because no-one can remember if Canadians have done anything retarded)
Duke Posted April 28, 2010 Posted April 28, 2010 *suspects SLMHC may be a merkin in disguise* (loads of US backpackers - IME - pretend to be Canadian, because no-one can remember if Canadians have done anything retarded) Naw, he said aboot, that's all the convincing I needed...
SLMHC Posted April 28, 2010 Posted April 28, 2010 I figured I'd loose everything if I wiped and recreated the RAID level. Better to ask a stupid question than to do something stupid. I have heard many a story of Americans slapping a Canadian flag on their backpacks as they travel Europe, and yes I am a born and bread Canuck. Oh and we really don't say aboot, but I can admit to letting out an eh or two.
apaton Posted April 28, 2010 Posted April 28, 2010 My best practice basic overview. This is how I think about things today, well may change tomorrow!! Mirror If the environment requires small/random (<32k) I/O then Mirror is best. Database fit in to this area, also VMware with iSCSI. VMware's VMFS is usually random I/O (typically around 5-10 VM's per VMFS) Triple Mirror Never used, people fell hard done by when they loose 50%, never mind 66% RAIDZ & RAIDZ2 Ideal for large sequential reads/writes (>128k), eg. File Shares and backups images. RAIDZ3 Paranoid users only, or when 2TB drive are available in S7000. Ideal for Archive/Backup data. Dedup Best on 7310/7410 with at least 1 Readzillia (100Gb L2ARC) Test before use, seriously consider compression as an alternative Record (Block) size is important, smaller better dedup ratio, large better overall system performance. (I'm still learning what the optimal number is for this, if any) Don't use when performance is paramount. (i.e. Database) Ideal for backups and VM templates. iSCSi Lower latency for Random I/O - Oracle, MS Exchange and MS SQL (Physical or VM's) Use a MIRROR RAID level Synchronous write bias to Latency (Use SSD ZIL if available) Consider Records (block) sizes of 8k (Data/Client dependent) NFS Higher latency, but good for sequential reads/writes. Ideal for Linux home directories VMware for general machines and VDI. RAIDZ2 Write Cache This caches writes to volatile memory, if you suffer a power failure you can loose data. The performance increase is very tempting, particularly when you don't have a Logzilla. Choices is yours. Andy 1
SLMHC Posted April 29, 2010 Posted April 29, 2010 Very interesting read Andy. I currently have a 7110 which hosts a mix of Windows and Ubuntu VMs (XenServer). I've got things like WSUS, SAV, Fog, and Sharepoint running on this gear and it generally runs just like on bare metal. My Exchange 2007 box though is laggy when accessing it and I do see the odd disconect when using Outlook. I'm using a NFS share to host all the VMs, with the default share settings. I also am using an iSCSI LUN on my SQL box to back up the DBs to it. Again, with the default settings. I have a dedicated Procurve assigned to SAN duties, and use LACP to create a 3 NIC datalink to the switch. Unfortunatly XenServer does not use Jumbo frames, so I cannot enable that feature. We are moving into a new building in the fall and I've asked for, and been given the go ahead, on getting a 7310 w/ a 4400 with 11 x 1TB disks and 1 x 18GB logzilla. The existing VMs will migrate over to this box (the 7110 will be repurposed to user drives and backup). In addition I will be building a XenApp farm, which will utilize the 7310. Any future servers will be built on to this gear. I beleive I have enough space for the near term and enough expandability for the long term. I have to upgrade my SQL server this summer and was going to put it on hardware as the sluggishness of Exchange scared me away from using the 7110. Would the 7310 w/ Logzilla be enough to virtualize it, given all that I hve going onto it? Should I be looking at a Readzillla as well? Or multiples of both? My budget it set but I may be able to get some things changed up, or purchase through my operating budget. I mention above that I use default settings for all my shares/LUNs. This is because I've never seen a whitepaper, forum post, or blog on what tweaks would be best to use depending on the usage. Your post shines some light on it for me, but, is anyone aware of up to date data on the subject?
andy0789 Posted April 21, 2011 Posted April 21, 2011 I use the 7110 as well. I have not had much luck with it as far as write latency goes with my VMs. I did have a Citrix XenApp farm running off of it for about 2 days until I had to move it to local storage on the XenServers, ruining my HA, because of the horrible latency the users were experiencing. I primarly use NFS with the exception of a couple Exchange datastores using iSCSI. Network utilization is low and I am also using LACP/trunking. We are also looking for a new NAS/SAN solution. I am interested in getting the 7320 and moving the 7110 to a remote site making use of the remote replication functionality to create an off-site disaster recovery location. I think I screwed the pooch with the 7110 since the beginning choosing Double Parity Raid for the pool, which I think killed my write performance. If I were to use this in a DR scenario and change the pool to RAID 10/Mirror, you think I could pull some decent performance out of this thing for around 15 VMs, 4 of them being terminal servers, 2 SQL servers, 1 Exchange server, and other various small VMs. I'm not looking for native performance since it would be a DR scenario but usable performance. Depending on when during the month the DR happens the SQL servers are not that hard hit. Love all the features of the 7110, but so far write latency is a big problem for me.
apaton Posted April 21, 2011 Posted April 21, 2011 Double Parity and Wide RAIDZ zpool are very poor for Random I/O. I agree by recreating a mirrored zpool and enabling write cache use will get a significant performance increase and reduced write latency. Plus look at Filesystem aliment, especially with SQL/Exchange on Windows 2003. (see http://www.edugeek.net/forums/hardware/43403-sun-storage-7000-series-articals-blueprints-3.html#post538278) Latest firmware. Andy 1
andy0789 Posted April 27, 2011 Posted April 27, 2011 Just as a followup for the 7110, I switched one of my virtual machines from NFS to iSCSI with write cache enabled and have seen significant speed improvements. VMWare was reporting the following averages before and after the switch (taken from the same time period on seperate days): NFS Datastore 140ms Write Latency 80ms Read Latency iSCSI Datastore 14ms Write Latency 34ms Read Latency Still could do better but atleast it is an improvement. Changing the pool to Mirror instead of Double Parity Raid could only increase performance. 7110 on 2010.08.17.2.0,1-1.18
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now