Update our virtualisation hardware stack, but to what?

[limawhiskey]

So over the next five years I can see us scaling back on site services and moving to education cloud offerings like G Suite or Office 365, but for now we need to renew our virtual server and backup infrastructure that was new in 2010 but is now out of warranty.

 

The question is, what should that hardware look like in 2017?

 

Currently we have two hosts and a SAN with backup to another building and duplicated to tape. We have around 1,500 students and 180 staff. On the infrastructure we're running a dozen VMs, mostly Server 2012 R2 with the usual network services. In addition we have:

• SQL (for SIMS).
  
• vCenter for VM management.
  
• Terminal server (2003 soon to be updated to 2012 R2), for at most 30 seats of remote desktop / app access, in place mainly for staff to use SIMS remotely.
  
• Cashless catering.
  
• And the odd low demand Linux server for various services (e.g. Xibo running on Ubuntu for signage).
  

 

So nothing too crazy.

 

Cost estimates to run these VMs in the cloud indicates an 'as-is' cloud move is a non-starter, and reliance on internet connectivity is not very desirable. Changing/improving ICT is not a focus for SMT here so they are not wanting to change the way it works for staff and students at the moment, so a full move to Office 365 and/or G Suite is a no go in the short term, although we do use G Suite for email.

 

Spec'ing this sort of infrastructure kit is not my area of expertise so I have asked a few suppliers for their proposed refresh designs to set the ball rolling. I haven't been too enamoured with the results. Across the board I have basically had an update of the current setup; two or three hosts and a SAN (either direct attach or iSCSI), with backup covered by a staging server / NAS then dupe to tape via an autoloader.

 

My bone of contention with this design is that I identify the SAN as single point of failure (see the King's College case - and that was a lot more high-end than our shared storage will be). We don't have the budget to go down the replica/redundant SAN route. Therefore a SAN based design doesn't improve on our current setup, nor is it actually better than a design of one (suitably spec'ed) host running all our VMs on local storage. In fact, you could argue it's worse as it introduces more complexity into the system.

 

My ideas are, in the order that I perceive to be least to most expensive:

1. Single host with solid backup/replication strategy. Disaster downtime limited to next business day hardware replacement and restore time.
   
2. Two hosts; one main host and another replicated host in a ready state for failover. A new stand-by host might be seen as a 'waste' in this design, so could utilise the old hardware on a temporary/best effort basis, falling back to the single host plan above if necessary.
   
3. Two or three hosts in a 'hyperconverged' cluster configuration, eliminating the SAN SPOF (e.g. StarWind, Scale Computing).
   

 

I'd also consider backup to cloud (e.g. Redstor), assuming we could demonstrate that costs over time are roughly comparable to buying all the new hardware.

 

A minor complication is that we're running RM CC4.5 but at this level of abstraction it shouldn't matter what is running the guest VMs. It would be easiest if we can use VMware to migrate the VMs as they are, but I'm not opposed to changing hypervisor.

 

So, those of you who are thinking about this sort of project currently, how are you designing your systems?

Edited March 10, 2017 by limawhiskey

[Linfit]

Have you considered a Dell VRTX ?

We moved from a stack of Dell Poweredges and an EMC SAN onto one of these two years ago and I think its the best thing I've ever bought.

Lots of options for how you want to spec it.

[Linfit]

Have you considered a Dell VRTX ?

We moved from a stack of Dell Poweredges and an EMC SAN onto one of these two years ago and I think its the best thing I've ever bought.

Lots of options for how you want to spec it.

[limawhiskey]

Have you considered a Dell VRTX ?

Thanks for the reply. Yes, I considered it after I saw one in a colleague's school a few weeks back and they were very happy with it too.

 

However in terms of design it's still a single point of failure. I think we can improve on that design within the same budget, or at least not pay over the odds for a design that does have a single point of failure.

[limawhiskey]

Have you considered a Dell VRTX ?

Thanks for the reply. Yes, I considered it after I saw one in a colleague's school a few weeks back and they were very happy with it too.

 

However in terms of design it's still a single point of failure. I think we can improve on that design within the same budget, or at least not pay over the odds for a design that does have a single point of failure.

[dhicks]

So, those of you who are thinking about this sort of project currently, how are you designing your systems?

 

We are in a very similar position, having hardware from around 2010 - it's trundling along okay, and disk capacity and speed is okay, but at some point it's going to need refreshing. We do have the advantage of a good Internet connection, though - 200mb/s, with capacity up to 1gb/sec should we need it. Hence, my plan is to start replicating functionality off in the cloud. For the moment, that's going to be a VPN gateway and a domain controller to get started with and see what monthly costs are going to be like (I vaguely guess somewhere under £50 for two quite small VMs). Those two server will always be running. Also, at some point, we should ditch the DC and move to Azure cloud Active Directory or whatever, but I'm still working up to that.

 

We'll move shared file areas to Google Team Drives, mirrored to local harddrives, and we'll start moving some backups on to Google Drive. We can then start adding further VMs, but just as fail-over server in case of on-site failure. This means they shouldn't need to be running the whole time, we'll maybe have them start up a couple of times a week to run updates. That's an MIS server (our MIS produces backup files hourly, so on startup the failover machine just needs to get the most recent backup from Google Drive), a file server (mounting Team Drives via Streaming Files).

 

We should also be able to export VM images from our on-site system (XenServer) to a format that AWS will import. We'll do that for our print server and our licensing / AV / updates / etc servers - a backup image taken once a week should be enough.

 

My very rough guess is around £200-£300 in monthly hosting costs. If we feel we need to add additional local capacity then I plan to look at second-user servers, which should be very capable for what we require them for.

[dhicks]

So, those of you who are thinking about this sort of project currently, how are you designing your systems?

 

We are in a very similar position, having hardware from around 2010 - it's trundling along okay, and disk capacity and speed is okay, but at some point it's going to need refreshing. We do have the advantage of a good Internet connection, though - 200mb/s, with capacity up to 1gb/sec should we need it. Hence, my plan is to start replicating functionality off in the cloud. For the moment, that's going to be a VPN gateway and a domain controller to get started with and see what monthly costs are going to be like (I vaguely guess somewhere under £50 for two quite small VMs). Those two server will always be running. Also, at some point, we should ditch the DC and move to Azure cloud Active Directory or whatever, but I'm still working up to that.

 

We'll move shared file areas to Google Team Drives, mirrored to local harddrives, and we'll start moving some backups on to Google Drive. We can then start adding further VMs, but just as fail-over server in case of on-site failure. This means they shouldn't need to be running the whole time, we'll maybe have them start up a couple of times a week to run updates. That's an MIS server (our MIS produces backup files hourly, so on startup the failover machine just needs to get the most recent backup from Google Drive), a file server (mounting Team Drives via Streaming Files).

 

We should also be able to export VM images from our on-site system (XenServer) to a format that AWS will import. We'll do that for our print server and our licensing / AV / updates / etc servers - a backup image taken once a week should be enough.

 

My very rough guess is around £200-£300 in monthly hosting costs. If we feel we need to add additional local capacity then I plan to look at second-user servers, which should be very capable for what we require them for.

[stevewill6]

I recently updated our server infrastructure to a visualized environment. We went down the route of 2 hosts with local storage and hyper v replication between them. Each server runs half of our vms and each half is replicated to the other host. We then have a backup server running Veeam. All the backups are copied to a NAS in another building and a copy taken off site.

 

Servers were HP DL380gen9, we also have capacity to add storage and ram if needed in the future.

 

Fairly cheap, very simple and so far effective

[stevewill6]

I recently updated our server infrastructure to a visualized environment. We went down the route of 2 hosts with local storage and hyper v replication between them. Each server runs half of our vms and each half is replicated to the other host. We then have a backup server running Veeam. All the backups are copied to a NAS in another building and a copy taken off site.

 

Servers were HP DL380gen9, we also have capacity to add storage and ram if needed in the future.

 

Fairly cheap, very simple and so far effective

[limawhiskey]

We went down the route of 2 hosts with local storage and hyper v replication between them. Each server runs half of our vms and each half is replicated to the other host. We then have a backup server running Veeam. All the backups are copied to a NAS in another building and a copy taken off site.

That sounds like a good design. I take it you have to manually power up the VMs in the result of a failure of one host? Did Hyper-V replication provide some features that Veeam replication couldn't?

 

Servers were HP DL380gen9, we also have capacity to add storage and ram if needed in the future.

So you went for fewer, larger disks and the same with DIMMs?

 

Fairly cheap, very simple and so far effective

Fits my requirements nicely

[limawhiskey]

We went down the route of 2 hosts with local storage and hyper v replication between them. Each server runs half of our vms and each half is replicated to the other host. We then have a backup server running Veeam. All the backups are copied to a NAS in another building and a copy taken off site.

That sounds like a good design. I take it you have to manually power up the VMs in the result of a failure of one host? Did Hyper-V replication provide some features that Veeam replication couldn't?

 

Servers were HP DL380gen9, we also have capacity to add storage and ram if needed in the future.

So you went for fewer, larger disks and the same with DIMMs?

 

Fairly cheap, very simple and so far effective

Fits my requirements nicely

[localzuk]

We have gone down the route as follows:

 

2 x Dell PowerEdge servers as hosts

2 x QNAP NAS's for storage - all SSD storage

 

The plan will be to have half our VMs on each host, with one of the QNAP NAS's as the primary storage for the node, then Veeam replication between the 2.

 

Its the most cost effective solution we could come up with without needing 2 clusters of Hyper-V Nodes.

[localzuk]

We have gone down the route as follows:

 

2 x Dell PowerEdge servers as hosts

2 x QNAP NAS's for storage - all SSD storage

 

The plan will be to have half our VMs on each host, with one of the QNAP NAS's as the primary storage for the node, then Veeam replication between the 2.

 

Its the most cost effective solution we could come up with without needing 2 clusters of Hyper-V Nodes.

[stevewill6]

Yes if a host failed i would have to manually fail over to the second host. 15 minutes of downtime was acceptable considering the budget.

 

We went with a single CPU in the servers as we dont need alot of processing power at this time but we have the option to expand in the future as the number of vms grows. We have 8 2.5 inch sas drives and a empty bay that will fit another 8 and 64gb of RAM. I am hoping to get a minimum of 7 years out of the servers as the previous servers we replaced were all 10+ years old, but they dont build them like they used to.

[stevewill6]

Yes if a host failed i would have to manually fail over to the second host. 15 minutes of downtime was acceptable considering the budget.

 

We went with a single CPU in the servers as we dont need alot of processing power at this time but we have the option to expand in the future as the number of vms grows. We have 8 2.5 inch sas drives and a empty bay that will fit another 8 and 64gb of RAM. I am hoping to get a minimum of 7 years out of the servers as the previous servers we replaced were all 10+ years old, but they dont build them like they used to.

[mavhc]

We are in a very similar position, having hardware from around 2010 - it's trundling along okay, and disk capacity and speed is okay, but at some point it's going to need refreshing. We do have the advantage of a good Internet connection, though - 200mb/s, with capacity up to 1gb/sec should we need it. Hence, my plan is to start replicating functionality off in the cloud. For the moment, that's going to be a VPN gateway and a domain controller to get started with and see what monthly costs are going to be like (I vaguely guess somewhere under £50 for two quite small VMs). Those two server will always be running. Also, at some point, we should ditch the DC and move to Azure cloud Active Directory or whatever, but I'm still working up to that.

 

We'll move shared file areas to Google Team Drives, mirrored to local harddrives, and we'll start moving some backups on to Google Drive. We can then start adding further VMs, but just as fail-over server in case of on-site failure. This means they shouldn't need to be running the whole time, we'll maybe have them start up a couple of times a week to run updates. That's an MIS server (our MIS produces backup files hourly, so on startup the failover machine just needs to get the most recent backup from Google Drive), a file server (mounting Team Drives via Streaming Files).

 

We should also be able to export VM images from our on-site system (XenServer) to a format that AWS will import. We'll do that for our print server and our licensing / AV / updates / etc servers - a backup image taken once a week should be enough.

 

My very rough guess is around £200-£300 in monthly hosting costs. If we feel we need to add additional local capacity then I plan to look at second-user servers, which should be very capable for what we require them for.

 

So does moving from 10/1Gbps dedicated server connections to 200Mbps shared with everyone using the internet, and 1ms ping to 100ms ping make the system better? You're saving what, £5 of electricity per month?

 

If all your clients are offsite, I see why cloud is a good idea, but how is uploading all your print files to the internet and downloading them again sane? I'm confused

[mavhc]

We are in a very similar position, having hardware from around 2010 - it's trundling along okay, and disk capacity and speed is okay, but at some point it's going to need refreshing. We do have the advantage of a good Internet connection, though - 200mb/s, with capacity up to 1gb/sec should we need it. Hence, my plan is to start replicating functionality off in the cloud. For the moment, that's going to be a VPN gateway and a domain controller to get started with and see what monthly costs are going to be like (I vaguely guess somewhere under £50 for two quite small VMs). Those two server will always be running. Also, at some point, we should ditch the DC and move to Azure cloud Active Directory or whatever, but I'm still working up to that.

 

We'll move shared file areas to Google Team Drives, mirrored to local harddrives, and we'll start moving some backups on to Google Drive. We can then start adding further VMs, but just as fail-over server in case of on-site failure. This means they shouldn't need to be running the whole time, we'll maybe have them start up a couple of times a week to run updates. That's an MIS server (our MIS produces backup files hourly, so on startup the failover machine just needs to get the most recent backup from Google Drive), a file server (mounting Team Drives via Streaming Files).

 

We should also be able to export VM images from our on-site system (XenServer) to a format that AWS will import. We'll do that for our print server and our licensing / AV / updates / etc servers - a backup image taken once a week should be enough.

 

My very rough guess is around £200-£300 in monthly hosting costs. If we feel we need to add additional local capacity then I plan to look at second-user servers, which should be very capable for what we require them for.

 

So does moving from 10/1Gbps dedicated server connections to 200Mbps shared with everyone using the internet, and 1ms ping to 100ms ping make the system better? You're saving what, £5 of electricity per month?

 

If all your clients are offsite, I see why cloud is a good idea, but how is uploading all your print files to the internet and downloading them again sane? I'm confused

[limawhiskey]

It would work well as a proof of concept and cost benchmarking project. I assume you can pull the guest back to a local one again if costs get out of hand.

 

I'd actually be really interested in the findings as the only estimates I managed to get were from a supplier that specialised in cloud services.

 

It didn't make sense when we looked at moving our services; the estimate over just three years far exceeded the cost of hardware to run a small cluster of VMs in house.

 

Clearly internet is not close to LAN for latency but gigabit connectivity is now pretty affordable.

[nicholab]

Have you considered using datacore in front of your current SAN and a second direct connected storage system?

[dhicks]

If all your clients are offsite, I see why cloud is a good idea, but how is uploading all your print files to the internet and downloading them again sane?

 

Goodness, no - sorry, I wasn't clear enough about the above setup, the above is simply for fail-over purposes in case of on-site hardware failure. Things like file servers and print servers will still be onsite - we might look at moving the MIS server permanatly off to the cloud, but we'll see how we get on just with a mirror server first.

 

I'd be interested to see if Google's new File Streaming, or any other similar systems, cache any data for read / write purposes locally - I can't see how file access for multiple clients would be practical otherwise.

[mavhc]

Goodness, no - sorry, I wasn't clear enough about the above setup, the above is simply for fail-over purposes in case of on-site hardware failure. Things like file servers and print servers will still be onsite - we might look at moving the MIS server permanatly off to the cloud, but we'll see how we get on just with a mirror server first.

 

I'd be interested to see if Google's new File Streaming, or any other similar systems, cache any data for read / write purposes locally - I can't see how file access for multiple clients would be practical otherwise.

 

Thanks for clarifying, I agree that all your services should be ready to move anywhere, run on your local cloud, easy to move.

[gshaw]

On-site Hyper-V with replication to Azure seems a nice hybrid setup at the moment. You can use hyper-converged hosts with SSD acceleration to avoid the SAN issues too.

[limawhiskey]

You can use hyper-converged hosts with SSD acceleration to avoid the SAN issues too.

Is that with native Windows / Hyper-V options or some third party integration?

[limawhiskey]

Have you considered using datacore in front of your current SAN and a second direct connected storage system?

No, I haven't heard of that product, just having a look into it now. Do you have any idea of rough costs for it?

[PotNoodleTech]

I recently updated our server infrastructure to a visualized environment. We went down the route of 2 hosts with local storage and hyper v replication between them. Each server runs half of our vms and each half is replicated to the other host. We then have a backup server running Veeam. All the backups are copied to a NAS in another building and a copy taken off site.

 

Servers were HP DL380gen9, we also have capacity to add storage and ram if needed in the future.

 

Fairly cheap, very simple and so far effective

 

This is what I have done as muy budget is consistently tight. This is probably the best value way you can do it. Two high spec hosts with filled with solid state drives and 5 year warranty replicated to each other and a solid backup system using a separate server using Veeam replicated to a NAS ina different building and preferably either tapes/secondary nas on separate site/cloud is all most schools need. The only issue for the OP is that he is currently VMware and would need to move to hyperv which will complicated things compared to staying with VMware.

 

OP - VMware do an outstanding bit of software called VSAN - you use 3 physical hosts each filled with disks and VMWARE shares out all disks on all hosts as one big fault tolerant pool, you can lose a disk, you can lose a server, the more disks and servers you add the faster and more resilient your pool. Licencing is expensive however it is orders of magnitude better than a SAN.