RAID6 vs RAID10 failures (and hotspares)

[Steve21]

Morning guys,

 

Been delving in our replacement storage for this year and just been wondering a few bits. Apologies in advance if I'm being a lemon on any of these assumptions

 

Disk Failures

Obviously RAID10 "can" support a lot more disks going "if" it's the correct ones failing, but surely if 2 drives fail that are the same mirror that's the raid gone compared to RAID6 where even if two went it wouldn't matter?

 

e.g. RAID10 say you got 21 drives (for easier maths know odd doesn't work ), one fails leaving 20 disks, no real performance issues or anything but you got a 5% chance if another fails it'll be the mirror and knock off the array?

 

RAID6 same example, 1 fails, leaves 20 with poor performance during rebuild, but another fails and it's no issue for array 0% chance of failure unless 3rd goes.

 

Disks per array

Is there any real benefit of using the larger array if you're not struggling with IOPS?

 

Surely lets say 20 disks giving 20k iops, or 2x 10 disks giving 10k iops per array should factor the same if things are split nicely across them. e.g. staff file server on one, students on another etc etc

 

e.g. RAID10 array, 20x 1TB disk, 10TB usable.

RAID10 array, 2 arrays, 10x 1TB = 5TB per array, 10TB total.

 

And gives again less chance of failures in relation to the 2 drives failing above?

 

Could understand RAID 6 etc as you're using the extra disks, so 10 would only give 8TB etc vs 18TB on 20 disks

 

Hot Spares

I know there's always an argument with RAID and hotspares generally in the IT world (Just look at spiceworks for that lol), but personally I love them If there's no-one on site all the time, or not a big enough team to have someone who could always do the swap outs if needed.

 

But giving the larger number of disks etc within the array does anyone run additional hotspares over the normal 1/2? (or none at all if you're in that camp? Relying on coldspares etc )

 

 

Sure I'll have some more random thoughts later but yeah haha

 

Thanks,

Steve

[pcstru]

Yes with RAID 10 you can be taken out if the wrong 2 discs fail. There is no parity so a rebuild is quick and a hot spare or two IMO makes sense (but then so does a whole mirror array)! With RAID 6, think very carefully about the performance hit during a rebuild. It will be a time when you are stressed and want the vulnerability window to be as small as possible, you may want to take a backup in addition to normal loads. Anything you do will slow everything down. Whether I'd carry a hot spare would depend on a number of things : disc size, number of spindles and what the array is being used for. The more critical it was, the less I'd want a hot spare in there but if it was that critical, I'd probably not want RAID 6. For IOPS it does depend on the load - particularly how peaks loads are being generated. Generally, more IOPS on a single array will be better placed to handle transient peaks although normal loads will be higher. Split the 20 discs into two arrays of 10 and your normal loads are less but your ability to handle peak load is halved.

 

So, IMO - it all depends!

[Steve21]

Yes with RAID 10 you can be taken out if the wrong 2 discs fail. There is no parity so a rebuild is quick and a hot spare or two IMO makes sense

 

For IOPS it does depend on the load - particularly how peaks loads are being generated. Generally, more IOPS on a single array will be better placed to handle transient peaks although normal loads will be higher. Split the 20 discs into two arrays of 10 and your normal loads are less but your ability to handle peak load is halved.

 

So, IMO - it all depends!

 

Aye thought that seemed right with RAID10 but seemed bit silly lol, guess sadfully you can't have every requirement in one raid option haha

 

That's true, I guess even the average IOPS doesn't really take into account any spikes during the day hmm.

 

Thanks,

Steve

 

Aye

[DMcCoy]

RAID6 on a decent controller should be fast enough for most things, with 20 disks, I think 2 10 disk RAID6 arrays would be a good price/performance/protection compromise.

 

I find it odd that so many people want so much storage, but then only want to use 1/2 of it!

[Steve21]

RAID6 on a decent controller should be fast enough for most things, with 20 disks, I think 2 10 disk RAID6 arrays would be a good price/performance/protection compromise.

 

I find it odd that so many people want so much storage, but then only want to use 1/2 of it!

 

Aye, am wondering about that. RAID10 seems great and all, but the wastage on it seems excessive for a school imho

 

24x 1.2TB drives gives like 26TB RAID6 or 14TB RAID10, could just buy 2x smaller ones in RAID 6 and have even better redundancy/space

 

I am beginning to wonder if it's even worth replacing the SAN for only 2 HOSTs (They had lots of physical servers that were replaced just with 2 HOSTs before I joined) or whether it'd be better looking at getting more hosts with internal storage, and just using the replication within Veeam/HyperV as the "high availability" part of it.

 

4 HOSTs, 2 each end of school, with proper replication etc setup in place, has got to be more reduant than 2 HOSTs on single SAN at one end right

 

Steve

 

Steve

[Arthur]

or whether it'd be better looking at getting more hosts with internal storage, and just using the replication within Veeam/Hyper-V as the "high availability" part of it.

If you are thinking of going with Hyper-V it might be worth having a read of chapters 2-4 from Hyper-V Best Practices. An except of which is shown below.

 

Storage Overview

Storage for Hyper-V is not primarily about capacity; it is about performance. Stop thinking in terms of capacities such as Gigabytes and Terabytes of data and calculate in IOPS from now on when Hyper-V Storage options are on the table. Of course, you still need to make sure there is enough capacity available. However, these typically neither influence the design nor the costs on a larger scale since high capacity hard drives are available for a very low price. There are very different approaches for a virtualization storage design.

 

In older versions of Windows Server and other virtualization products, a NAS system for smaller environments and enterprise-grade SAN systems were used as a storage backend. With a Fibre Channel or iSCSI connection to the Virtualization Server Systems, it provides a central storage for all nodes in a cluster, ensuring capabilities such as VM live migration and Cluster Failover.

 

The loss of cluster nodes does not impact the integrity and availability of the storage systems. The Logical Unit Numbers (LUNs) of SAN/NAS systems appear as local drives on the Hyper-V hosts. Traditional architectures came with a one-VM-per-LUN approach, which was dropped in modern approaches for Cluster Shared Volumes that hosts multiple VMs per LUN. SAN systems of NAS systems for small Hyper-V setups are still the most often used storage architecture around these days and are fully supported with the recent version of Hyper-V. The use of Cluster Shared Volumes has been improved a lot and is the default deployment option for storage volumes for Hyper-V on a SAN.

 

A more modern approach is the use of the onboard capabilities of Windows Server storage management. Leveraging the SMB3 performance and availability options of a Microsoft Scale-Out File Server (SOFS) with attached JBODs allows you to generate IOPS for Hyper-V on a large scale for a considerably lower price. A Microsoft SOF allows you to utilize the best features you know from a SAN with just one or more physical server systems running Windows Server. You can use traditional storage LUNs behind a SOFS or use Storage Spaces with locally attached hard drives. These hard drives are connected through the system as just a bunch of disks (JBOD) without the use of a hardware RAID controller.

 

Availability on the storage level is done on a software level by Windows Server. Storage spaces and JBODs offer scalable performance; they are not just a small business solution. VMs are no longer placed on LUNs; instead, they leverage SMB3 file sharing. Continuously available file sharing allows the use of central File Server clusters with hardware resiliency. If you lose a File Server, the shares hosting Hyper-V VMs will not be interrupted. All VMs keep running, even those that were using the lost File Server just a second ago. A transparent failover kicks in to ensure availability of your Hyper-V storage.

 

The most cost-effective hardware solution behind a Windows File Server Infrastructure for SMB3 are standard Server systems with SAS-attached JBODs. There is no need for expensive SANs or even a RAID controller to achieve high availability and good performance with SMB3-based file services. Storage Spaces build a full storage virtualization abstraction of the connected disks that are / and is even capable of automatically tiering used SSDs and classic hard drives in the same JBODs for optimal performance.

 

Of course, Hyper-V can also leverage local hard disks in a Server system for the storage of VMs. The VMs can even be replicated to another Hyper-V host in terms of disaster recovery like you learned in Chapter 3, Backup and Disaster Recovery, but they do not offer High Availability capabilities and should therefore not be used in production.

 

http://a.pomf.se/qlotfh.png

 

SANs versus SOFS

One of the most crucial decisions in a storage design process for Hyper-V is the question of whether to stick with the traditional SAN model or hop on the wagon of Microsoft's SOFS Architecture. Both solutions can do the main job, that is, providing IOPS for VMs, without any problems. Having done many projects using both architectures, the following is some real-world guidance I use for storage design.

 

The first and most important design principle: do not use nonredundant systems in production for Hyper-V clusters—no local hard drives, no single hardware SMB3-Shares, and no single point of failures on your storage design. If you cannot fulfill these requirements, don't plan for High Availability on Hyper-V. Plan for disaster recovery instead. Uncluster the Hyper-V Servers and replicate the VMs between the nodes in a small setup or between smaller clusters in a bigger environment via Hyper-V Replica. Having said that, let's focus on the decision to make.

 

Technically, it's possible to use a SOFS in conjunction with a SAN. However, with SAN Vendors adapting SMB3 protocols, this is not an efficient long-term scenario in most cases and we will focus on using SOFS with JBODs and Storage Spaces. There are some SOFS architectures with a SAN you should leverage, particularly, for Fibre Channel SANs, where the Hyper-V hosts may only have Ethernet adapters. Leveraging the SOFS as the entry point to the storage environment reduces the configuration complexity of the "compute" nodes by placing all the storage vendor requirements on the "storage" nodes of the SOFS cluster. You can also place several SANs behind a single SOFS cluster and provide a consistent management and storage presentation experience across the virtualization platform, which will allow you to deploy any variety of storage vendor hardware.

[rich_tech]

RAID6 on a decent controller should be fast enough for most things, with 20 disks, I think 2 10 disk RAID6 arrays would be a good price/performance/protection compromise.

 

I find it odd that so many people want so much storage, but then only want to use 1/2 of it!

 

I went raid6 on our SAN with that in mind, rather spread things over both controllers than all in with an "it can go bang bigtime scenario".

 

I think the second point is because price wise SAS/Enterprise SSD disks are horrible which probably accounts for a lot of overspeccing at install times, plus its better to have too much space and sitting there unused than run out or run that risk, plus it does fill up over time with creep.

 

Our sites original NAS box which was used for media file sharing, has now crept up to one and a half times the disk size of its old home when it was copied to a new home after the original linkstation went bang.

Edited March 1, 2015 by rich_tech

[Steve21]

If you are thinking of going with Hyper-V it might be worth having a read of chapters 2-4 from Hyper-V Best Practices. An except of which is shown below.

 

 

Hmm certainly seems interesting option.

 

The short story behind this, started new job over Summer and they have an old old SAN that's 99% full, and 2 new HOSTs (HyperV clustered) with no built in storage on them. Which obviously currently means no redundancy at all if any issues with SAN/server room.

@Arthur I may be wrong but from what I was reading before you can't put SOFS on a host using HyperV can you? or has this changed? So assuming I'd need another machine to "serve up" the shares?

 

Long term wise looking to mirror server-room to our DR room on other side of school, so guessing this would effectively have a secondary SOFS on other side too? Same principle as SAN?

 

Thanks,

Steve

[Steve21]

I went raid6 on our SAN with that in mind, rather spread things over both controllers than all in with an "it can go bang bigtime scenario".

 

Silly question @rich_tech but are you using those 2 RAID6 as separate storage? Or are you effectively doing the mirroring yourself on the SAN? Just using the RAID6 as you get more space.

 

Thanks,

Steve

[rich_tech]

Our SAN is pure storage as a CSV set here in hyper-v, I backup all our VMs off cluster using veeam to the 2x NAS that we have (8 and 12tb of storage on the pair of them respectively, with a little bit of use for backing up the schools photograph archive)

[tmcd35]

RAID levels and design is a product of function not fashion. Choose the one that is right for the task at hand. RAID 5, 50, 6, 60, etc. are all perfectly sensible and acceptable solution depending on the scenario. As @DMcCoy, above, I cringe at the thought of spending ££££'s on 15k SAS drives and using RAID-1. The cost per Gb has to be factored in at some point. On the other hand if you are using 4Tb 5k SATA drives, RAID-1 is probably sensible.

 

These days though, I think hardware RAID is quickly becoming yesterdays technology. Now with have multi-core processors and cheap RAM software RAID tends to make more sense. Systems like ZFS and Storage Spaces offer greater flexibility and allow you to grow you storage array over time in a way that is harder to achieve in hardware.

 

Our current storage array is 2 drives in RAID-1 for OS, 12-drives in RAID-50 for storage, and 2 host spares. They are all 450Gb 15k SAS drives. It's fast, reliable, rock solid, and has been for nearly 5 years. We replace a couple of drives each year whether they need it or not, rebuilds take about 8 hours per drive. It was the right system design at the time for the budget we had for the project.

 

We are about to replace it with a system based on two replicated Windows Storage Spaces boxes with 1xSSD for OS, 2xSSD for cache, and a mixture of 15k 450Gb SAS and 10k 900Gb SAS for data. I'm pretty sure we'll be using parity storage spaces rather than mirrored storage spaces because I'm not convinced by the benefits of mirrored disks in this set and the increase in cost per GB is extortionate. I do have some spare 2Tb SATA drives that might go in some of the spare bays in one of the servers for things like video files, isos, etc. They'll probably be either straight stripped or mirrored.

 

So, the question is - what type and size drives are you using, and what data is being stored on the drives? Only with that information, plus possibly user count, could recommendations on which RAID scheme you use be meaningful.

[pcstru]

I find it odd that so many people want so much storage, but then only want to use 1/2 of it!

Our first tier storage only uses 1/4 of the raw storage capacity. For us that is about striking (trying to) an appropriate balance between resilience and performance.

[Steve21]

So, the question is - what type and size drives are you using, and what data is being stored on the drives? Only with that information, plus possibly user count, could recommendations on which RAID scheme you use be meaningful.

 

Currently nothing Looking at new storage for the network as old SAN here that's full to the max. Currently looking at 24x 1.2TB 10k SAS based SAN as the original item, but looking at alternatives too. Running HyperV cluster for up to 1k~ users

 

Already at 8TB used on current SAN, so just wondering whether using RAID10 on 28TB down to 14TB~ is cutting it too fine when I know there's a few TB additional to migrate from old physical servers still.

 

But then even £15k+ on a SAN could get a few new hosts with built-in storage etc.

 

Steve