Deploying Storage Spaces Direct For Virtual Machines

[FN-GM]

Hello,

 

I am looking at Deploying Storage Spaces Direct instead of a SAN to replace our aging Hyper-V setup. I was wondering if anyone has gone down this route and is able to share experiences?

 

Thanks

Edited March 27, 2018 by FN-GM

[snagrat]

Yes we did this.

Took us a while due to the hardware requirements and we had to buy extra parts in the end that were fully supported.

 

Performance is better than on our SAN previously

[FN-GM]

Was it easy enough to setup? Did you do it yourself? Did you follow any guides please?

[snagrat]

Was easy. I followed this guide https://argonsys.com/learn-microsoft-cloud/articles/storage-spaces-direct-configuration/

 

We don’t do VLANs so a lot of the networking bits were skipped

[pritchardavid]

Done this myself too.

 

4x Supermicro Servers

10x Intel 1.8TB NVMe in each

2x SATA DOM for Boot and DC on each

2x Mellanox 56GBE cards in each

2x Mellanox 56GBE Switch

 

Cost a lot, but no problem with speed for storage

 

Just wish we spent more on processors and more RAM

Wouldn't mind another node or two through, more fail-over and load balance then

 

Wasn't that bad to setup either......

I created a new VLAN myself, just for the servers to separate from the server from the network.

[browolf]

Do you have to use "Server 2016 Datacenter" edition?

[JackCPickup]

Was it easy enough to setup? Did you do it yourself? Did you follow any guides please?

 

PluralSight has some good vids on it.

[JackCPickup]

Do you have to use "Server 2016 Datacenter" edition?

 

Correct.

[browolf]

Correct.

 

But isn't the license for that super expensive?

[Katy]

But isn't the license for that super expensive?

We pay about £150* a year for each datacentre licence on our educational pricing, so not that bad considering we only need three to cover the three hosts plus unlimited VMs on them.

*might be more, might be less, can't remember but it's in the £100-£200 bracket

[snagrat]

But isn't the license for that super expensive?

 

The idea of Storage Spaces is to run Virtual Machines.

 

If running lots of VMs then Datacenter becomes the most cost effective license

[JackCPickup]

But isn't the license for that super expensive?

 

Not so bad for schools. Cheaper than most software defined storage solutions if you're a school.

[RaviTester]

We are a small school 300 pupils, working with a tech support company, and when I suggested S2D they said they hadn't heard of it in schools! I wonder if you could contact me with your experience. I would appreciate an input as we are considering a server upgrade over summer and I would like to hear about how you got on.

[email protected]

[Norphy]

If I were you, I think I’d be looking for a different tech support company.

[PotNoodleTech]

It looks really interesting in 2016. I tried storage spaces in its first incarnation and it was only fast if you used the mirroring feature and instantly lost 50% of your storage. SSD caching worked excellently though!

 

Looks like it is worth a go now if you can get datacentre licences on your EES subscription cheap!

 

So that example above is very overkill with 100GB switches and 4 x servers. I wonder what the minimum we can get away with is. I bet with 3 hosts and only 10GBe networking it will still be faster than most SANS and a darn site cheaper!

[PR-UK]

We deployed this back in September last year, so far so good. We have about 80% of our VMs now running on it. We bought specific equipment from Dell as part of our infrastructure refresh so this had all been tested with S2D for performance/compatibility. The only slight divergence I made from the standard instructions was to split the Cluster/Storage Traffic to two 10GB NICs and VM Traffic to two separate 10GB NICs.

 

My thoughts?

• Don't go for the minimum of two nodes, it wasn't supported in the preview builds but resellers campaigned for it so it was added in, however performance and reliability have been questionable and the vast majority of S2D failures I've read about are down to deploying two node clusters
  
• If you're reusing hardware keep in mind that S2D doesn't work on drives identified as RAID, if your storage controller only supports this mode then you'll need to replace it
  
• Check your networking capabilities, the NICs used for Storage replication should support RDMA to offload the processing to the NIC rather than the server CPU, there's other networking technologies that S2D can make use of to reduce latency and improve speeds make sure you know what they are, that your switches support them and that they are working
  
• Follow the guides, Dell, Lenovo and a few others have decent white papers on setting S2D up, read them even if you don't have their hardware as they go in to more depth than some of the MS documentation ("Just type Enable-ClusterS2D in PowerShell and everything just works!")
  
• Test, test, test. Use VMFleet to check performance of Storage Replication, test hardware failures by unplugging servers to make sure S2D can handle the failures
  
• Server Updates, use Cluster Aware Updating in Failover Cluster Manager to correctly deploy updates to each node, it will allow the time for the node to update, reboot, and re-replicate storage before taking the node out of Maintenance Mode and moving on to the next node. Compared to our old SAN based Windows Cluster which took ~90 mins to update 6 Nodes, the 4 Node S2D takes 5+ hours
  

 

Hope that helps!

[FN-GM]

Hello,

 

 

I am looking at brand new with Dell. They have said they will set it up for free as part of the deal. I will just need to stick my VM on when they are finished. Not bad!

 

Thanks for all the comments.

[mukz]

Hello,

 

 

I am looking at brand new with Dell. They have said they will set it up for free as part of the deal. I will just need to stick my VM on when they are finished. Not bad!

 

Thanks for all the comments.

 

Are they also providing you with the dell switch and Nics to go with?

[FN-GM]

Are they also providing you with the dell switch and Nics to go with?

 

NICS in the server yes. We are using our existing Cisco Switches. They are 10 gig, still being made and still under warranty.

[mukz]

Ah right okay. One of the schools i help sims with ended up getting the whole solution for about 25k. We are looking into the S2D too.

 

Maybe i can pick your brain next year when we actually get around to doing it.

[PR-UK]

We started with new Dell Servers, then decided to replace our HP Core Switches as they were quite old and didn't support all the functionality. Then we decided we wanted to get 10GB to the edge switches and the price HP were charging for 10GB GBICs it was only slightly more expensive to buy new Dell edge switches with all the 10GB GBICs we needed so we replaced ~80% of the edge switches to.

 

Couldn't fault Dell, they were quick to provide a quote, answered all our questions and put us in contact with their networking specialists and S2D people to answer questions, we even got a few days with a S2D test system in one of their data centres to see how it all worked.

 

I contacted HPE and Dell on the same day, I spoke to Dell who took all my requirements and used their DPAC service to evaluate our existing infrastructure. HPE called and weren't interested in quoting for S2D they just wanted to sell us a HP SAN or one of their HP Hyper-converged systems, I told them I was willing to look at them but wanted quotes for S2D servers to... I'm still waiting.