New Unified Storage Device

[Willott]

Right, so in the final stages of deciding and ordering a new device for storage. Opinions on the following ideas would be appreciated, I've had quotes for the big names (EMC, NetApp, expansion of our old HP SAN), and some smaller, newer unified devices (QSAN U600Q - comes in cheap, but their unified side of things seems quite new). Of course, budget is tight, but I want to try and get the following:

 

10GbE connectivity

Ability to saturate (or near saturate) 10GbE for sustained read and close to that for sustained write

High IOPs (thousands)

Low Latency

Serving CIFS/SMB clients, and NFS to Xenserver (or ESXi)

 

If anyone spots anything stupid with the above, please challenge...

 

Options I've been looking at are:

 

Nexenta server (Nexenta Gold licenced)

2x E5-2620 Xeon processors

128GB RAM

2x 500GB disks (syspool)

10GbE SFP+ network card

21x 1TB SAS2 7k2 RPM disks (main storage pool)

2x ZeusRAM (slog)

1x ZeusIOPS 200GB (L2ARC)

 

Self Build OpenIndiana with Napp-IT

2x E5-2620 Xeon processors

256GB RAM (ARC)

2x 300GB disks (syspool)

10GbE SFP+ network card

21x 1TB SAS2 7k2 disks (main storage)

2x ZeusRAM (slog)

1x ZeusIOPS 200GB (or 400GB L2ARC)

 

Obviously there is a cost difference due to licensing and slight difference in spec (and self build over built by company). The other option would be to virtualise the storage using either RDM or PCI-E Pass-through on ESXi

 

So use the above self build spec with either Nexenta or OpenIndiana (with a couple more 300GB disks in JBOD), give VM 192GB RAM, and have 50GB ish for running other VMs on the box (with 10GbE speed if not more). Later expansion would then be put a second head node in (processors, RAM, disks for ESXi), 2 SAS switches and have the JBOD as the only point of failure (which could then be overcome with replication to a similar setup elsewhere).

 

Now the issues I have - Nexenta don't fully support the E5 series of processors yet (even though they announced support before E5 was released), so not overly sure whether I want to go with them, also, during my conversations it's been mentioned of a few issues with version upgrades. OpenIndiana has a lot of community support, and I'm very happy in a command line environment and have a fair number of sources of help... but I'm concerned by the "mythical IT bus" aspect. Or do I just go with one of the major companies, sit back on my laurels, enjoy the fact I'm fully covered by support, but not have 10GbE (or ability to upgrade to it), no SSD caching and probably much lower performance than the above setup, or go with the QSAN, with support, 10GbE, SSD caching, but non mature software (the write cache is not currently available but will be in next firmware release).

 

Any views appreciated.

 

Cheers

 

Will

 

PS AAAAAHHHHHHHHHHHHH!!!!!!!!!!!!!!

Edited July 3, 2012 by Willott
added end scream

[gshaw]

Non-mature software and SAN aren't two phrases I'd want next to each other personally. Whatever you buy is going to determine the reliability and long-term stability of your servers \ data so I'd stick with something well known and supported.

[glennda]

Take a look at Dell Equallogic (its quite cool!) or for a self build look at Open E softwares.

[Willott]

Yep, that's why I'm shying away from the QSAN. The issue I have with well known and supported is that usually means vendor tie-in, extra cost for licenses as we move forwards, lower possible speeds (no SSD caching after all... or SSD support at all in the boxes I'm looking at).

 

Looked at a dell solution, however it seemed overly complex - had an iSCSI back end, then a 3U bundle to 2 heads and a battery unit for doing the NAS side of things... was less than inspiring, and to get the 10GbE I'd have had to stick in a new £10k switch.

 

Open-E I've used previously and didn't get on particularly well with it. May have a look to see if it's improved.

 

I know what I'm looking to do in the budget I've got is a blooming big ask, but one must try!

 

Cheers

 

Will

[glennda]

One thing i would say with what you have posted above is the main storage is on SATA - I would go for 10k SAS minimum.

 

Also If you are looking at Custom Build SuperMicro do a Massive 36 Bay chassis which is good for large disk arrays. I built a backup server with 16 2TB HDDs and lots of space for more!

[Willott]

Main storage is on SAS, just 7200RPM rather than higher speed. On SAS to not conflict with SAS expander. One other thing I'm currently limited by is rack space, I have a 2u gap and a 3u gap, so looking at 1u head with 3U 28 drive JBOD.

 

On the drive side of things, the drives will be either in RAIDZ3 or Mirrored VDEVs spanned in pool to get some performance out of them. The bigger performance increase should come from the ARC, L2ARC and ZIL (ARC and L2ARC should be enough to cache our working set, so once ARC and L2ARC populated, read contact with spinning rust will be minimal, write wise, ZIL should take care of ensuring that writes to the block storage is done in larger chunks, ie more sustained write type so reducing high IOP requirement from individual disks, and mirroring should see that IOPs are high enough to provide solid base). There's enough space in chassis to have 20 drives configured as mirroed vdev (RAID10 near enough) with 1 hot spare, the 2 slog/ZIL devices (8GB ZeusRAM), 1 L2ARC device, 2 syspool disks and have 2 bays spare (possible for expanding L2ARC).

 

I would think that, on the SMB/CIFS side of things, we tend to be more read heavy, and on the VM side of things (NFS), writes will hit ZIL (NFS does sync writes), so performance should be good (in fact, I had some suppliers suggesting sticking datapool as RAIDZ2 rather than Mirrored).

 

Cheers

 

Will

[gshaw]

Btw what are you aiming to run from the SAN? Have you done a capacity planner to check what level of IOPS you'll need first?

[Willott]

Currently we have a SAN in place which is struggling, so it's difficult to know the number of IOPs we need. I'm looking presently to provide SMB/CIFS access to 300 clients, and NFS access for a few VMs including a Smoothwall VM (which is hitting disks quite hard currently), a Securus VM (again, quite heavy on the disks), and other Windows and Linux VMs of lower load. However, in the future, I'd like capacity to be available to do VDI (or whatever technology they bring out that does VDI but better), so I'm not only considering current use, I'm looking at having some overhead for expansion.

 

Edit: Just looked at our current SAN and seeing 2000 IOPs currently reported by it (considering that's on the RAID6 7k2RPM array I'm impressed... though a little bemused as to how we're hitting that many IOPs on that... not sure it can physically handle that!) - just found Forefront Endpoint Protection stalled on a file... which may be what was causing that load.

Edited July 3, 2012 by Willott

[glennda]

2000 IOPS? I think the MSA2000 I used to have had a max of 38000 So one of them may work!

[Willott]

I was talking about theoretical limit for the disks... thought it was a max of about 100 IOPS for a 7200RPM disk

[gshaw]

2000 IOPS? I think the MSA2000 I used to have had a max of 38000 So one of them may work!

 

Max isn't the problem... how far your wallet can stretch to buy the disks to fill that capacity is another matter

 

It's ~80 random IOPS for a SATA disk, sequential can hit more but all depends on the workload being thrown at them (well that's how I've always understood it anyway)