Raid or Storage spaces, anybody taken the plunge yet?

[Jaan]

So we're toying with the idea of looking at server 2016 for our hyper-v servers in the not too distant future.

 

We plan to have OS c: as a Raid1 and the d: formatted as Refs with either a RAID 10 (or 50) config, but storage spaces is another option for the d:.

 

Anybody using this yet on a hyper-v? (or file server)

[CAWJames]

We are about to order a new server Dell 730xd with around 12TB of raw space, 96GB RAM, dual Xeons etc, and have been thinking of how to carve it up. Usually I would keep 2 drives in a RAID 1 for OS and RAID 10 the others. We will be using 2016, so we also have the question, do we go RAID-1 NTFS as normal for the OS(ReFS can't be a boot volume) and then storage space the other disks with ReFS and to hell with traditional RAID or go the usual route.

 

I have tested ReFS on my test ring for Hyper-V and the creation of big disks has gone from a long lunch break to literally seconds, and they say there are good speed boosts in overall running of Hyper-V over ReFS, plus no CHKDSK(woo hoo).

 

So ReFS is definitely the file system of choice, just now have the RAID vs SS argument to have in my head! Software RAID has never been MS's forte!

 

So I'm with the OP, is SS mature enough for production? Is ignoring years of RAID mentality and just going JBOD on the controller and using software a good thing? Any caveats or mines in the way?

 

This is going to replace our 7 year old R710 DR Hyper-v server, so does nothing apart from backups 99.9999% of the time, until the worst happens and it has to handle our critical production VM's until the cluster/building/town/county/[insert DR scenario here] is fixed/rebuilt/replaced.

 

Thanks James

[Fez]

We have just installed two of the 730xd's in the same config using two 480gb ssd's for boot in raid 1 and then used Windows Server 2016 Storage Spaces to create a cluster. Only issue we had was it not accepting raid drives but a small powershell tweak and this went away (We couldn't get the control to report non raid even when set as JBOD)

 

So Far its been working perfectly for us and the performance is a huge leap from our old R710's

 

Feel free to drop me a message if you need any help as ive already gone through all of the struggles.

[CAWJames]

Thanks Fez, what are your storage spaces drives? We have the option of nline 7.2 or 10k, and torn between the 2

[Fez]

I went with the following on the spec to give you an idea.

 

Intel Xeon E5-2650 V4 2.2GHz 12C/24T X 2

96GB Ram DDR4

2x 400GB SSD

2x 1.2TB 10K RPM SAS 12GB Drives

3x 4TB 7.2K SATA 6GB Drives

12 Bay Chassis

 

We went this line so that we can expand the RAM And storage should we need to in the future. I went with two 10K drives to be the hot storage and then using the others as the cold storage and SS seems to handle this well so far. As we use our for live VM's Im in the process of rebuilding our old ones to be other nodes that we can use for backup and further cold storage.

[CAWJames]

Much the same as the spec we are looking at, but 2 x 300GB 15K 12GB 2.5" SAS for OS and 6 x 2TB 7.2k 12GB 3.5" NLSAS, as this is a DR server, it will be doing DPM backups most of the time, and only have to actually run productions VM's in an emergency, so we thought NLSAS should be fine across the board with faster 15k for the OS, the SSD's look nice but expensive(unless they actually aren't, I didn't get a quote for them), if we were upgrading our production VM Cluster, then I would probably spec SSD.

[DMcCoy]

Do keep an eye on ReFS if you are using DPM less than 2016, as 2012 R2 doesn't (or didn't, not looked for a while) support backing up VMs from a ReFS volume. DPM 2016 *should* be ok now though. I'm not sure about Backup Exec, i'd given up on ReFS by that point, especially as pre 2016 you can't use one of it's useful features (integrity) with vhdx files.

[steve]

Interesting blog on why not to use consumer grade SSDs in Storage Spaces Direct: https://blogs.technet.microsoft.com/filecab/2016/11/18/dont-do-it-consumer-ssd/

[Jaan]

I went with the following on the spec to give you an idea.

 

Intel Xeon E5-2650 V4 2.2GHz 12C/24T X 2

96GB Ram DDR4

2x 400GB SSD

2x 1.2TB 10K RPM SAS 12GB Drives

3x 4TB 7.2K SATA 6GB Drives

12 Bay Chassis

 

We went this line so that we can expand the RAM And storage should we need to in the future. I went with two 10K drives to be the hot storage and then using the others as the cold storage and SS seems to handle this well so far. As we use our for live VM's Im in the process of rebuilding our old ones to be other nodes that we can use for backup and further cold storage.

 

Hi Fez,

 

I have a question if that ok.

 

Are you using any ssd as cache disk for the s2d or are they all mechanical? I was under the impression that at least two cache drives (ssd) were needed for a s2d volume.

 

cheers

[SHimmer45]

Interesting blog on why not to use consumer grade SSDs in Storage Spaces Direct: https://blogs.technet.microsoft.com/filecab/2016/11/18/dont-do-it-consumer-ssd/

 

interesting article!

[Sagima]

We have 7 servers using refs now (one hyper-v host, two fileservers and 2 DPM 2016 backup servers).

 

I have had no luck inn getting S2D to work with the fileservers. They fail the setup test on sector size even though it's identical on both servers. The cache drives I tried to use were SSD.

 

ReFS seems to be a bit quicker when dealing with small files and creating really large ones like vhdx files but otherwise doesn't seem to have much performance increase on anything else.

 

It all backs up fine with DPM 2016, I may have backed up the virtual server when I still had dpm 2012r2 as I think I can remember installing a CU letting it back up ReFS

[Blue_Cookeh]

We have 7 servers using refs now (one hyper-v host, two fileservers and 2 DPM 2016 backup servers).

 

I have had no luck inn getting S2D to work with the fileservers. They fail the setup test on sector size even though it's identical on both servers. The cache drives I tried to use were SSD.

 

ReFS seems to be a bit quicker when dealing with small files and creating really large ones like vhdx files but otherwise doesn't seem to have much performance increase on anything else.

 

It all backs up fine with DPM 2016, I may have backed up the virtual server when I still had dpm 2012r2 as I think I can remember installing a CU letting it back up ReFS

 

Interesting that you chose ReFS on your file servers, I value dedupe far too much.

[Sagima]

I imagine it will come eventually. After we passed 7TB and seemed to be adding 10-30gb a day on our main deduped fileserver my manager had me go on a space reclaiming spree, going through and deleting old stuff, resizing pictures, emptying recycle bins and we got our undeduped files down to a mere 4.5TB so you can imagine how much stuff we had on them.

 

What I miss is DFS replication as I can't get S2D to work currently and seems much better (and as I leave in 57 days I may never).

[Fez]

Hi Fez,

 

I have a question if that ok.

 

Are you using any ssd as cache disk for the s2d or are they all mechanical? I was under the impression that at least two cache drives (ssd) were needed for a s2d volume.

 

cheers

Hi,

 

I didn't go for SSD cache as the OS is running on SSD's. We mainly use ours for virtual machines which don't get that much use and we store all other files on a NAS so no real need for fast cached storage. You can create the volume using any drives you want and it will handle how it splits it if you have multiple drive types.

[SHimmer45]

this is something we are possibly looking at for this coming summer i did put a post up asking for people's experiences and what hardware they have been using (what JBOD enclosures etc) but havent had much response.

[Jaan]

this is something we are possibly looking at for this coming summer i did put a post up asking for people's experiences and what hardware they have been using (what JBOD enclosures etc) but havent had much response.

 

We'll be looking at recon dell 730XDs with 24 front 2.5" and two rear mounted 2.5" for the server OS

 

 

also S2d this might be an issue with this server... unless a firmware update resolves it.

 

Even when set to Jbod, it fails the s2d compatibility test. However the thread below says there's a "workaround"

 

http://en.community.dell.com/support-forums/servers/f/906/t/19660614

Edited January 16, 2017 by Jaan

[ColinP]

Do all the hosts have to be identical?

[Jaan]

Do all the hosts have to be identical?

 

from what i understand, only from a storage standpoint.

[PotNoodleTech]

I looked into storage spaces and did some testing and benchmarks. It is good. The SSD caching works great, it's fast and it works as intended. You lose 50% of your storage as it's basically a mirror. I ended up going with raid as I was using SSD drives only so felt raid was actually more flexible than storage spaces as aside from the SSD cache you can set up things with a balance of speed, redundancy, cost in any mixture. Also RAID is proven technology storage spaces was a bit new for me to jump on for production. Having windows software control the storage array sounds like it could be problematic. Also it may have improved since I last looked at it on Win2012r2.

[ColinP]

I dont think its quite the same as 2012 as this video dosent suggest its a raid 1 equiverlant

 

https://www.youtube.com/embed/raeUiNtMk0E

[Jaan]

Awesome articles from Microsoft here:

 

Fault Tolerance:

 

https://technet.microsoft.com/windows-server-docs/storage/storage-spaces/storage-spaces-fault-tolerance

 

Understanding Cache:

 

https://technet.microsoft.com/en-us/windows-server-docs/storage/storage-spaces/understand-the-cache

 

How it stores your data:

 

https://blogs.technet.microsoft.com/filecab/2016/11/21/deep-dive-pool-in-spaces-direct/

 

Storage spaces storage calculator:

 

Storage Spaces Direct Calculator

[PotNoodleTech]

I dont think its quite the same as 2012 as this video dosent suggest its a raid 1 equiverlant

 

https://www.youtube.com/embed/raeUiNtMk0E

 

Ok that vid is really cool as I suspected 2016 has raised the game again!

 

So there's two options, mirroring and parity. Mirroring is similar to RAID 1/10, Parity Similar to Raid 5/6. On 2012 parity was unusable as the write performance was atrocious. I am guessing (hoping!) they have addressed this with 2016 as several servers in parity would be the ideal way to go as long as it performs well.

 

Cheers for all the info ColinP and jahilton2002

[markm75]

I just spotted this thread.. i have a similar question over at this thread.

 

I'm still trying to get a handle on storage spaces.. i've looked at the pages related to it.. do you not need a controller card to leverage it.. unsure how a 24 bay chassis with port expansion (12gb sas) would work exactly, maybe a port off the motherboard.. and on fault tolerance.. how can you do a raid 6 equivalent on one single server.. in our case its for DPM 2016.

[SHimmer45]

my understanding is that you need a HBA card to attach your server to a external disk array or internal disks in the actual server (HBA card needs to support certain standards mind you)

storage spaces will allow you to do mirroring and parity and can be setup to "allow" multiple disk failures

[Jaan]

As a test i used a dell t610 with a perc6/i controller. It doesn't support pass-through (jbod) only raid levels. I tried to to configuire each of the 8 disks as RAID0, which did present them to the OS. However storage spaces basically laughed at me and said "Nice try Jnr". It was just as a test, i would never do that in production.

 

So a true Jbod controller is needed. However they also need to be server 2016 (S2D) certified. We were looking at the dell r730xd servers. Although the H730 controllers support jbod, they aren't certified so fail the S2D tests. However looking at this post on dell, there seems to be a work around: Dell R730xd with H730 Mini and Non-RAID/HBA mode and W2016 Storage Spaces Direct - PowerEdge HDD/SCSI/RAID Forum - Servers - Dell Community