Hyper-V 2SD network setup

[andy_nic]

Hi all,

Looking at creating a Hyper-V Cluster with Storage Spaces Direct, one thing i'm kind of stuck is how the network is best setup in terms of Virtual switch to Physical Nic Fibres 10GBps to physical network switch. Some say to Team all together and separate at virtual switch, some say Team and give each team a role like management, storage, etc.

 

Andrew

[notalot]

Hey, I have several Hyper V clusters with S2D setup (3 and 4 node clusters), from my experience it works both ways without too much issue, both ways pass the cluster validation tests.

 

My current setup is to have 2 10gb NIC's teamed for Cluster Communications (and S2D server to server coms) and the rest of the NIC's teamed for Virtual Switches and management. I vlan off the Cluster communication then leave the rest tagged for all the other vlans that are needed.

[kevin_lane]

Interesting I’m looking into setting this up

 

Would you be willing to go more into detail on your setup or provide guidance

[ColonelPanic]

Hi Kevin,

 

In a quick nutshell

We have a 3 node cluster here with S2D set up. There a 2x2 10GB NICs in each node. We have a direct network between each node, and then one of the ports on each NIC goes to the core switch (and backup core switch).

We then run a 1GB line into the core switch for iLO.

 

I think I still have the setup guide for S2D, I'll see if I can dig it out.

[Jaan]

I made this pretty pics once..... same as @ColonelPanic really..... we ended up with a switch instead of interconnects that the pic shows.

 

[notalot]

Hi, below is the config of one of my sites, not much difference in a 3 node cluster between direct attached or via a switch we just set them all up the same but in a 4 node cluster you would need quite a few additional NIC

 

[ATTACH=CONFIG]72995[/ATTACH]

 

My normal config across the sites is

 

Dell Poweredge R650xs

2x Intel® Xeon® Silver 4314 CPU @ 2.40GHz

128gb Ram

6x 2.4TB 15k SAS Drives

2x 980GB SSDs'

4x 10tb SFP Ports (normally needs 2 network cards but can get away with 1)

 

If your building this up your self make sure you get a HBA Card and not a Raid card as S2D needs direct access to the drives to work.

 

Assuming that your not using these for Azure Stack HCI then there is also no actual requirement for the certified hardware. The difference between this cost wise was mad, we got it quoted last year £7.2k per server that we got and it was £36k per server (Dell-AX-640)

 

One small difference that I would do now is to have an internal NVMe for the OS drive rather than using one of the 2.4tb drives.

 

The actual setup of this is very straight forward at a high level

 

• Install OS
  
• Team Nics
  
• Set IP address
  
• Join Domain
  
• Enable Remote Desktop (if you permit this)
  
• Run below Powershell (I'm sure the PS can be improved to do more just didn't get round to it in the deployment schedule)
  
• Create Virtual Switch in Hyper-V
  

 

# Fill in these variables with your values
$ServerList = "Server1", "Server2", "Server3"
$FeatureList = "Hyper-V", "Failover-Clustering", "Data-Center-Bridging", "RSAT-Clustering-PowerShell", "Hyper-V-PowerShell", "FS-FileServer", "RSAT-Clustering-Mgmt", "rsat-hyper-v-tools"
$Clustername = "Clustername"
## Set line 33 for the IP of the cluster (the command never liked setting the ip via a variable)


# This part runs the Install-WindowsFeature cmdlet on all servers in $ServerList, passing the list of features into the scriptblock with the "Using" scope modifier so you don't have to hard-code them here.
Invoke-Command ($ServerList) {
   Install-WindowsFeature -Name $Using:Featurelist
}

#This tests all disks and resets them if something has been written to them or a partition has been created, this doesn't effect boot drives

Invoke-Command ($ServerList) {
   Update-StorageProviderCache
   Get-StoragePool | ? IsPrimordial -eq $false | Set-StoragePool -IsReadOnly:$false -ErrorAction SilentlyContinue
   Get-StoragePool | ? IsPrimordial -eq $false | Get-VirtualDisk | Remove-VirtualDisk -Confirm:$false -ErrorAction SilentlyContinue
   Get-StoragePool | ? IsPrimordial -eq $false | Remove-StoragePool -Confirm:$false -ErrorAction SilentlyContinue
   Get-PhysicalDisk | Reset-PhysicalDisk -ErrorAction SilentlyContinue
   Get-Disk | ? Number -ne $null | ? IsBoot -ne $true | ? IsSystem -ne $true | ? PartitionStyle -ne RAW | % {
       $_ | Set-Disk -isoffline:$false
       $_ | Set-Disk -isreadonly:$false
       $_ | Clear-Disk -RemoveData -RemoveOEM -Confirm:$false
       $_ | Set-Disk -isreadonly:$true
       $_ | Set-Disk -isoffline:$true
   }
   Get-Disk | Where Number -Ne $Null | Where IsBoot -Ne $True | Where IsSystem -Ne $True | Where PartitionStyle -Eq RAW | Group -NoElement -Property FriendlyName
} | Sort -Property PsComputerName, Count

#Lets test the cluster, warnings are generally ok especially if you are segerating the networks
Test-Cluster -Node $ServerList -Include "Storage Spaces Direct", "Inventory", "Network", "System Configuration"

# Pause to let you check the results.
Pause

New-Cluster -Name $Clustername -Node $ServerList -NoStorage -StaticAddress 1.1.1.1

Enable-ClusterStorageSpacesDirect -CimSession $Clustername

Add-ClusterScaleOutFileServerRole -Name SOFS -Cluster $Clustername

#this creates a new virtual disk for the cluster to use, its had some issues for me so the GUI can do this stage as well so its commented out for now
#New-Volume -FriendlyName "Volume1" -FileSystem CSVFS_ReFS -StoragePoolFriendlyName S2D* -StorageTierFriendlyNames capacity

#This sets the cluster failure behaviour without this if a host dies the cluster wont fail it over for 120 seconds.
(Get-Cluster).ResiliencyDefaultPeriod = 0

 

Once the PowerShell is done the cluster should be active as well as S2D, if you didn't have luck with the new-volume command then you need to create a Virtual Disk.

 

I will say that I have installed 5 of these and all 5 have had different issues after install, most related to permissions in DNS or AD for the cluster and the hosts.

 

Hope that lot helps a bit.