Disaster recovery - recover to Azure?

[localzuk]

I'm now in the process of rewriting our slightly out of date disaster recovery procedure for our network, and have come to the point where I am not sure which direction I should go with the recovery procedure.

 

We backup everything with Veeam, then weekly upload that to Azure archive tier storage - so in the case the building burns down we have that data.

 

Now with recovery, we have the option of downloading it all (it is split into 4 tiers of importance) and rebuilding it back on hardware on-prem. This means getting replacement hardware asap - which can take a day or 2, along with power and internet connectivity.

 

The other option, is recovering to Azure. We'd have a VM sat in Azure, spun down, which can mount the Azure blob storage, and allow us to restore the data direct into Azure. This would be considerably faster - we wouldn't need to download 11TB of data so we could get everything up and running very quickly. Including a RDS setup accessible via VPN. We could then later migrate each VM on to the new on-prem servers as we got them.

 

To run our setup like this in Azure looks like it'd cost us around £5000 a month, so we'd not want to run it too long like this.

 

What are others doing? Would you go for the super quick, but expensive option, or go for the slower but cheaper option?

[mavhc]

What's the main part of the 5k? Having 11TB online?

[RobFuller]

I'm in the exact same position as yourself @localzuk re-writing our DA plan and trying to work out the best "temporary" recovery location for our data.

Cloud hosting makes the most sense and TBC to get such a quick recovery in a total disaster I don't think £5k per month would be a balking point for us, given this would only be a temporary arrangement until on-site facilities were ready to takeover.

[localzuk]

What's the main part of the 5k? Having 11TB online?

Simply running the VMs, and a VPN to access them. Basically, it'd be spinning up our entire infrastructure in Azure.

[pete]

IIRC you're providing central services for multiple schools, right? Which skews it more towards "pay money to get things up and available".

 

In terms of costs, what does your insurance policy say about spend on stop-gap services to maintain availability? If it was a flood (for example), then a flood cleanup company's bills would be covered by the insurance.

[aguacate]

Personally I think that the disaster recovery plan needs to take into account the level of disruption, the scenario you point to at the top assumes that the school is operational but the IT system is completely wiped, that is unlikely but worth writing down, i guess what Im trying to say is that you could plan for the cloud restore of critical services which will likely be less than 5 grand for however long it takes for normal operation to resume.

[dmj]

Have you thought about deploying them as 'infrastructure as code '? This would mean that you only back up the data and not the servers themselves. To recover on site you would run your install scripts which would provision your VM's and copy the data back. You would have to get your virtual infrastructure working first, and it will be a big job to rewrite every config in every server as code. Of course, containerisation would save a lot of time and money here. You could have a service mesh from onsite to offsite clusters and have no downtime when your building burns - but from a previous thread I know your not keen on that - but it does seem the obvious solution here.

[localzuk]

Have you thought about deploying them as 'infrastructure as code '? This would mean that you only back up the data and not the servers themselves. To recover on site you would run your install scripts which would provision your VM's and copy the data back. You would have to get your virtual infrastructure working first, and it will be a big job to rewrite every config in every server as code. Of course, containerisation would save a lot of time and money here. You could have a service mesh from onsite to offsite clusters and have no downtime when your building burns - but from a previous thread I know your not keen on that - but it does seem the obvious solution here.

Pretty much none of this would be possible with a normal Windows network like ours - Windows network services don't live in containers, Microsoft doesn't support that. Eg. Windows DHCP, DNS, file server, etc...

[dmj]

Pretty much none of this would be possible with a normal Windows network like ours - Windows network services don't live in containers, Microsoft doesn't support that. Eg. Windows DHCP, DNS, file server, etc...

So put them in containers.

docker pull mcr.microsoft.com/windows/servercore

[localzuk]

So put them in containers.

docker pull mcr.microsoft.com/windows/servercore

I'm still struggling to see the advantage of running a Windows network in containers. If someone fancies writing something up saying why it would be better than simply running VMs, like most of us do now, that'd be great. As it stands, it seems to just be adding a layer of complexity.

[mavhc]

So you can run them all on one VM and save £5k?

[dmj]

I'm still struggling to see the advantage of running a Windows network in containers. If someone fancies writing something up saying why it would be better than simply running VMs, like most of us do now, that'd be great. As it stands, it seems to just be adding a layer of complexity.

 

Running 50 VM's costs you £5000 per month in Azure. If you can run your 50VM's on 5 Container hosts it will cost more like £500 per month and automatically scale as needed. Suddenly the costs of a cloud migration are much more realistic, plus you get self healing etc.

The reason is that VM's require a minimum of 1 vCPU (often 2) and a bunch of RAM. Containers can be given as little as 1% of a cpu, which means we can have a LOT of applications per server - when the applications grow, we power up more containers and more servers, when they shrink we shrink them. There are a lot of advantages here.

[localzuk]

OK, so I'll play along for a bit.

 

We have the following services running:

 

2 x AD DCs - running AD, DHCP, DNS, Google Sync and Office 365 Sync tools

2 x NPS radius servers

1 x Print server

1 x Antivirus proxy appliance

1 x Antivirus VM scanning consolidation appliance

1 x Veeam server

1 x certificate services server

2 x file servers

1 x RDS license server/KMS server

4 x RDSH servers

1 x RD connection broker

1 x Zabbix server

1 x SAML server

1 x 3CX server

1 x RDS gateway/web access server

2 x SCCM servers (1 is the database, the other runs the other services)

2 x IIS web servers

1 x WSUS server

1 x Xibo server

6 x SIMS servers

 

There's some other legacy stuff but it isn't important.

 

You're saying that all that will use less resources as containers than it does as VMs? Even when VMs can scale down to very small levels - eg, our DCs currently are using 1GB RAM each, and an entire 80Mhz of CPU (I still find the VMWare way of measuring CPU usage odd, haha). A VM for each of those comes in at about £8pm. (The big cost ones are the remote desktop servers and SIMS servers)

[dmj]

Yep. If you can probably run it on a single machine

Most kubernetes clusters will require a minimum of 3 servers though.

Check it all runs on windows server core and start building !

[dmj]

oh word of warning: Azure doesn't support Kubernetes on Windows yet, support will come - but for now you will need to run in GCP and wait for microsoft to get their act together. I suspect it will be there by summer.

[Boredguy]

Not sure Core would be wise for the SIMS servers, as don't think Solus 3 likes it without a GUI, but you never know.

 

I'd go with spinning up some servers in Azure to get the core features running, but leave RDP etc til the new hardware arrives

[localzuk]

The thing with RDP is that it gives device agnostic instant access to a working environment for staff. Those that don't have laptops etc... So its somewhat key to the whole DR process.

[mavhc]

Good news is you won't need your WSUS server if the school is closed

[localzuk]

Good news is you won't need your WSUS server if the school is closed

Recovery is getting everything back up and running again. So, getting temporary accommodation in place, including client devices - which, yes, will need to be kept up to date. It isn't in our "critical services" tier, so wouldn't be spun up day 1, but it would be recovered at some point.

[Boredguy]

The thing with RDP is that it gives device agnostic instant access to a working environment for staff. Those that don't have laptops etc... So its somewhat key to the whole DR process.

I guess it would depend on which of your users currently use the RDP day to day. I might be that instead of an instant DR of having full RDP provision up and running on Azure, that a scaled down version to those critical users that need it while sourcing the new hardware could be a compromise.

 

If you had said it was just students that used the RDP, then it's a service that may be worth delaying recovery. But if it was admin staff who use it to access SIMS, then that is a more important requirement.