New Servers (hooray) Hyper-V Configuration

[Warwick_Tech]

Hi All,

 

As a prequisite we have 4 physical servers, traditionally I always believed a DC couldn't be virtualised, because the host would need to connect before the DC so therefore would cause issues (?)

 

However I now know that's not the case;

 

So my new theory is to purchase two new virtual hosts and have them replicate to each other, IE the whole cluster would run off 1 VM, then replicate to the second - Or split the servers in half, but replicate the other half onto the second if that makes sense.

 

Anyway, using this article https://docs.microsoft.com/en-gb/archive/blogs/canitpro/step-by-step-virtual-machine-replication-using-hyper-v-replica was Microsofts typical rabbit hole, so I thought I'd get some real world opinions.

 

-----

 

Realistically how would you rate this as a solution?

 

Would you still keep some physical servers (DC, FS, SQL etc.) due to their demands both space, power and availability or are we now at the point where hyperV is stable enough to cope with running a system in tandem over two virtual hosts?

 

-----

 

I am keen to hear how you have your setups and what (if any) failovers you have

[Primus]

Run everything virtually. I wouldn't use replication I'd establish a cluster either with some shared storage or with S2D.

[Warwick_Tech]

Run everything virtually. I wouldn't use replication I'd establish a cluster either with some shared storage or with S2D.

 

I thought of this, but then introducing shared storage then creates a single point of failure

[slugshead]

What you should really do is setup High availability failover clustering rather than replication since it's such a small scale using some shared network storage, Synology NAS boxes are more than capable of doing iSCSI so that's a cost effective option.

 

The VM's can "float" between the two servers, you can live migrate the VMs between the hosts, allowing you to do maintenance and there's redundancy.

 

It's also scalable, so you can add more hosts/storage at a later date.

 

I have our 60 VMs spread across 3 hosts in this manner, including 2 DC's. But I do have a physical server running as a third DC for emergencies, that's located in a different building.

 

As for backups, Veeam takes care of everything

[slugshead]

I thought of this, but then introducing shared storage then creates a single point of failure

 

 

You have two routes (So two switches) and your storage has everything doubled (power/controller) with multiple connections (4-8 typically) and disks are in a raid array.

[Primus]

I thought of this, but then introducing shared storage then creates a single point of failure

 

Not necessarily!

[Aprice]

Previous site I worked at had a 3 host S2D cluster.

Each host ran a DC for the hyper V domain in standalone Hyper-V which stayed on that host- i.e doesn't failover.

 

HV1 - HVDC1

HV2 - HVDC2

HV3 - HVDC3

 

Then the main network DCs ran on top of the failover cluster, free to move between hosts.

 

Cluster 1

SCHOOLDC1

SCHOOLDC2

SCHOOLDC3

 

 

S2D worked well in the end, and will be rolling it out and maintaining it at other sites, you can do a two node direct connect cluster as well which works nicely, you just need a NAS or some storage share off the cluster I normally suggest a Synology box which can also run Synology Active backup or hold the Veeam data.

 

I think S2D would be the best solution in your instance, the VHDs for all your production VMs sit on the Cluster Shared Volume which is replicated constantly between the hosts. Replication is much slower (relatively) probably better suited to multi-site situations.

 

Not sure there is really a need to have separate physical servers now, don't have a problem with people running a physical DC though if you have spare budget or a half decent spare server.

Edited December 10, 2021 by Aprice

[ThatBoringBloke]

Keep it simple.

 

Have at least one physical DC.

Have two HyperV Hosts that run the live VM's

Have a third Host that the other two replicate to.

 

No need to over complicate.

[smarties11]

Run everything virtually. I wouldn't use replication I'd establish a cluster either with some shared storage or with S2D.

 

I thought of this, but then introducing shared storage then creates a single point of failure

 

IMO shared storage via a traditional SAN is overkill and too complex for an edu environment. This is where we have come from, on a 3 node VMWare platform. It's also more expensive, the SAN solutions we were quoted were more expensive than S2D by some margin.

 

Our new platform is a 2 node Hyper-V cluster with Storage Spaces Direct (S2D) as the storage layer. If you're not familiar with S2D, it's worth a look - it's essentially a vSAN/software defined storage - local storage from each server is pooled into a singled shared entity. In this scenario, you split your VM compute across the two hosts for performance/load balancing, and use the pooled storage for the VHDX storage. The storage is resilient across your nodes - so in a two node solution, it is mirrored, think software RAID 1. The storage network side of things in a two node solution is a direct RDMA connection between the two servers, in our case 2 x 25GB so 50GB bandwidth total. With more nodes, you require an RDMA switch. Failover and resiliency running the nodes as a Hyper-V cluster means that you can lose the entire node (or as many disks in ONE node as you like) and the other node will take over the compute for all your VMs and things continue to run seamlessly (so you need to spec accordingly in terms of RAM and cores). Additionally, we also have our S2D volumes set as nested mirror-accelerated parity - which means we can also tolerate one disk failure in the remaining node if one node is down.

 

The beauty of S2D is caching. You can spec relatively lazy SATA 7,200rpm storage drives and then cache these with NVME. If you get your cache ratio's right, it will perform as well as an all flash SAN or S2D cluster in day to day operations.

 

S2D is well worth a look. We've run it since October in production and it's awesome. The performance is fantastic, and much cheaper than continuing down the SAN route. I'm really happy with our decision and the solution we have now is far more resilient than what we had before.

 

One thing to bear in mind is the cluster quorum. It requires a 'witness' to establish which node is the most up to date in the event of any issues. There are various options for this, we we use a file share on our backup NAS.

 

Cluster and S2D setup is quite PowerShell heavy, especially for some of the finer options. For that reason we decided to buy in third party engineering for this, choosing a company who have done many of these setups before. It's the first time in my career I've used third party support for server config, always doing it myself, but it was worth it IMO. Just make sure you still understand all the basics so you can support it moving forward.

 

If you're looking at backups at the same time, I'd highly recommend Altaro combined with NAS storage and cloud storage via Wasabi. I did a post on it here -> http://www.edugeek.net/forums/security/223075-backup-strategy.html -> saved us a fortune on our old setup and again more performant and more resilient.

Edited December 10, 2021 by smarties11

[Warwick_Tech]

Hi All,

 

So a cluster and S2D seems to be the optimum solution; also based on these articles

 

https://www.veeam.com/blog/what-is-hyper-v-failover-cluster-windows-server.html

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview

 

Thanks for your input all, and of course still welcoming new input

[smarties11]

Hi All,

 

So a cluster and S2D seems to be the optimum solution; also based on these articles

 

https://www.veeam.com/blog/what-is-hyper-v-failover-cluster-windows-server.html

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview

 

Thanks for your input all, and of course still welcoming new input

 

Just to add; I'm not sure who your preferred vendor is when it comes to servers, but we engaged directly with Dell and they arranged for me a 2hr conversation with one of their top techies in Europe, who gave me an absolutely amazing overview and live demonstration of S2D. It really helped me to visualise, understand and feel secure with the technology. He even offered to give me VPN access to his test datacentre to have a play! We didn't go that far as were already happy enough with what we had seen.

[Jaan]

We have A physical DC-01

A VM DC-02

3 S2dNodes

 

Veeam backs it all up, all Dell hardware

[Warwick_Tech]

Hi all,

 

Raising this from the dead as I'm getting closer to this project -

 

So far I have decided:

 

* x1 Physical DC (Old DC will remain in place)

* x2 S2D nodes covering the other HyperV servers with a link cable between them

 

The old DC becomes the 'witness' and I'm going to build a second DC on the S2D cluster.

 

My only issue with this setup is no one can give me a definitive answer that when you do remote desktop or physical connections to the two S2D nodes, whether you need two RDP sessions and two IP addresses, or do they essentially act as a singular unit.

[Jaan]

Hi all,

 

 

My only issue with this setup is no one can give me a definitive answer that when you do remote desktop or physical connections to the two S2D nodes, whether you need two RDP sessions and two IP addresses, or do they essentially act as a singular unit.

 

I rdp into my nodes servers the same way i do anything else.

 

This is done via the "management" nic interface. The nodes behave as if they are two separate machines. Its the S2D that does the magic.

 

You normally manage Virtual machines (on non s2d) server via the hyper-v manager.

 

On a s2d cluster, you can create a vm via hyper-v manager as you would normally....... but if you want to add it to the cluster (s2d), you "add" a role via the "failover cluster manager" service running on the clusters.

 

The role detects the vm running on the server as "standalone" (in hyper-v). Once you've added the vm as a role, it is added to the s2d cluster. From that point, you can manage the vm from the "failover cluster manager" as it is part of the cluster.

 

TL:DR

You can run a vm in hyper-v as standalone on any given physical node server. Once you add that vm as a role, its clustered and part of s2d.

 

edit: think i kinda repeated myself above..... my nodes have two seperate IPs used for management tasks.

Edited February 15, 2022 by Jaan

[Warwick_Tech]

I rdp into my nodes servers the same way i do anything else.

 

This is done via the "management" nic interface. The nodes behave as if they are two separate machines. Its the S2D that does the magic.

 

You normally manage Virtual machines (on non s2d) server via the hyper-v manager.

 

On a s2d cluster, you can create a vm via hyper-v manager as you would normally....... but if you want to add it to the cluster (s2d), you "add" a role via the "failover cluster manager" service running on the clusters.

 

The role detects the vm running on the server as "standalone" (in hyper-v). Once you've added the vm as a role, it is added to the s2d cluster. From that point, you can manage the vm from the "failover cluster manager" as it is part of the cluster.

 

TL:DR

You can run a vm in hyper-v as standalone on any given physical node server. Once you add that vm as a role, its clustered and part of s2d.

 

edit: think i kinda repeated myself above..... my nodes have two seperate IPs used for management tasks.

 

This makes sense -

 

also watching some video bits which helped

 

And a long and boring Microsft Docs

https://docs.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-direct-overview

 

My other worry was complete disaster recovery, but It does happily state it can all be restored via Windows Server Backup (Other options are avaliable)

[dcwhitworth]

Personally I'd be very tempted to get some consultancy on this. Even if I was reasonably confident of what I was doing, it's such a critical system to get right that I'd want to at least have someone who did this sort of stuff regularly to be available, if not actually do it themselves.

[KK20]

I can tell you what I did and the pitfalls of what I wanted and tried to do. Firstly we run 4 servers in a stretch cluster with 2x SANS.

 

SVR1 + SVR2 + SAN 1 is in "one location" and SVR3 + SVR4 + SAN 2 is in another location. I use 10gb as network interlink between them. The stretch cluster is set as synchronous with a separate server as a witness. The separate server is a cheap R430 and is a physical DC (with a virtual DC in the cluster). SVR3 and SVR4 do not get a vote in the cluster, that way if I have a failed interlink SVR3 and SVR4 will not "split brain".

 

Originally I had SVR1 + SVR2 + SAN1 in a cluster, I added the stretch part "live", there was no downtime. I have had a failure in SAN1, the primary controller borked in a way that locked the SAN, the second controller did not take over. This produced a file lock in the cluster and "redirected" storage took over in SAN2 - in essence, it all worked. I also have had interlink failed (truck + fibre = fibre loses), split brain did not happen.

 

The hardware is roughly similar, 1 and 2 are R630, 3 is an R630 but 4 is an R640. All my VMs can run on a single server if necessary. SAN1 is an svc2020 with a mix of flash and HDD. SAN2 is an aging DAS powervault 3220 with a mix of flash and HDD. SSD must be used for your cluster log.

 

Now for the gotchas I found. Originally I had the powervault 3220 as the solo SAN. Originally I wanted to leverage S2D but hit a few snags. First, Microsoft is picky, really really picky as to the setup of the server running S2D, you will need a specialist to make sure your hardware supports the configuration. My "old R630" could not be made to S2D. Your mileage may vary but I had issues. Ideally I would have liked a pair of svc2020 with replication licenses and let the SANS replicate to each other thus I would have simply had a 2 or 3 way cluster and a pair of self replicating SANs. My budget wouldnt stretch to this and I would never have had the money to add a SAN plus some new servers. I also didnt have the budget to buy 2x fully capable S2D servers (thus "binning" our existing servers + SAN) so S2D was out for me.

 

stretch cluster will not work with "presented" drive mixed cluster sizes - take this into account if you are using large capacity drives and small capacity SSD caches - the clusters might be different - look for 512e at least. The QNAP threw 4k as a presentation REGARDLESS of what drives were in the system. This meant you would not have been able to have the QNAP as a stretch cluster drive for a SAN presenting 512e.

 

We have an 10Gb QNAP, it is a TS-1273U RP. It is a decent NAS with good performance. It is loaded with ironwolf drives and an SSD cache - great for iscsi backups. Bad for stretch cluster, the iscsi is just not reliable enough - the IOPS were decent enough, there were quite a few dropouts when I initially tried to set up a test cluster using the QNAP as a testbed. I would not have like to put our VM infrastructure on it!

 

As it was, I could keep the infrastructure I had, "stretch it" with a pair of new servers plus new SAN and now I have resiliency. Not only that but I can have a fire in one building and not lose my infrastructure! If SVR1 + SVR2 were to permanently be out of commission, I would need to issue a powershell command to allow SVR3 + SVR4 a vote on the cluster, 1 + 2 would be evicted at this point and would need to be "re stretched" back.

 

Stretching 7Tb of data makes very interesting task manager data and does look good when the network card is hitting 10gbs, it took all weekend and basically throttled the VMs right back. You can artificially throttle replication speed with a powershell command, otherwise your network interlink WILL be saturated, obviously more 10Gb cards and lines is the solution if you can afford this. Our interlink is a failover pair 10Gb link at the switch level, it is not LAGd - iscsi MPIO and LAG is not a good idea so I didnt want to try Microsoft stretch and LAG.

 

Backup is done from the hypervisor using veeam. we used to have a "cheap" QNAP file backup to quickly backup and restore user files, this was conducted from a VM. We did this because VEEAM is p*ss slow restoring when all you require are a few files. The QNAP restore was much quicker for a "I deleted my file". The QNAP is pretty much redundant now we are in onedrive and sharepoint (it actually backs up FROM sharepoint and onedrive now using boxafe, but that is a different story)

 

Advantages of stretch cluster over S2D - Hardware is less picky, you can have completely assymetrical hardware - I do have assymetrical servers!. if you are reusing old equipment and cannot do a complete swap then stretch cluster is better. You are relying on an older set of hardware for part of your resiliency though. that being said SANs have their own resiliency built in. Expanding an S2D cannot be done without downtime, I believe you shut the whole things down and bring back up with the new drives, I could be wrong on this though! I do know that symmetry has to be perfect and im not sure how mixing cluster sizes will work. SANS are at the mercy of whatever drives you can throw in them.

 

 

IF I had the money I would go with a 3 way S2D identical hardware with SSD+HDD journal/storage. For 2 nodes I would definitely nest (reducing capacity further)

Edited February 24, 2022 by KK20