Downtime Allowances - DfE Standards for Servers and Storage

[Ditto]

There's so much I could say in this thread, but I'll stick with one point, for now.

 

Given the requirements and all of what has been said, doesn't it add weight to the idea of taking as much as possible to the a reputable cloud provider.

 

You of course need the cloud provider to meet all the requirements that are in their remit, but any decent one should meet them anyway. Besides, the are likely to be able to utilise economy of size that most schools could never match. Clearly there are still on site requirements, internet access, wifi etc, but it should be a lot less than the alternative.

[msi_school]

1 VM per service was fairly sensible 10yrs ago but nowdays is just a massive waste of resources, especially if the VM is over provisioned in the first place to account for periods of high usage. I guess you can do some vertical scaling here, but it's probably still out of reach for single server schools.

 

For sure, but some of the larger MAT's approach the size of mid-level universities, who absolutely do re-architect services and attempt to run them in efficient ways.

 

I guess it's just one of those things that the more services you run the cheaper each service is.

 

Sorry could you expand on this, what is a better option than one VM per service? I realise that it is wasteful to have computing power that is not used and that each VM takes a percentage of the available power to run the OS however, I always assume that this overhead is minimal and the flexibility that this gives with uptime and servicing is worth the cost.

[enjay]

1 VM per service was fairly sensible 10yrs ago but nowdays is just a massive waste of resources, especially if the VM is over provisioned in the first place to account for periods of high usage.

 

Agreed. We have a whole bunch of things taking data out of SIMS (Locker, Wonde, Go4School, Azure AD sync, Google sync, ParentMail, SchoolCloud, Admissions+ and no doubt more), it would be expensive to have a separate VM for each of those tasks, especially since each service only runs for a handful of minutes every day.

[TechMonkey]

I would count them as the same task, so a sync server doing all those. I tend to go VM per task, not service.

 

I guess the next level would be to containerise each service and run in some sort of Docker implementation?

[msi_school]

I would count them as the same task, so a sync server doing all those. I tend to go VM per task, not service.

 

I guess the next level would be to containerise each service and run in some sort of Docker implementation?

 

@enjay is correct you can go too far in splitting services per VM but what I was asking was what is considered best practice now if this isn't.

 

I would generally split things by importance, user and task so I would for instance have the following servers:

• Student data
  
• Staff data
  
• SIMS
  
• Cashless Catering
  
• Print Server
  
• MDT
  
• Sync Services (Locker, Wonde, Go4School, Azure AD sync, Google sync, ParentMail, SchoolCloud, Admissions+)
  
• 
  

 

This separates roles and services between things that need to be up all the time form others that can be a bit more flexible.

[dmj]

Sorry could you expand on this, what is a better option than one VM per service? I realise that it is wasteful to have computing power that is not used and that each VM takes a percentage of the available power to run the OS however, I always assume that this overhead is minimal and the flexibility that this gives with uptime and servicing is worth the cost.

 

I guess the next level would be to containerise each service and run in some sort of Docker implementation?

 

Yes, this. A kubernetes cluster to run the container images, there's very little overhead from each server and the applications scale horizontally across the cluster.

This is what we run now, I also set up the same at a Uni I worked at - but never in a school.

[psydii]

Yes, this. A kubernetes cluster to run the container images, there's very little overhead from each server and the applications scale horizontally across the cluster.

This is what we run now, I also set up the same at a Uni I worked at - but never in a school.

 

I appreciate you are not quite suggesting school techs should set up a kubernetes cluster and port their school's LoB apps to run in containers.

 

The new platform is the cloud, where the SAAS runs on the PAS and that is what is running on kubernetes or equivalents.

 

Meanwhile back in legacy-on-prem in the SME space... we're running Windows, AD, SIMS, various MS SQL based LoB & back-end apps, with a load of file shares and printers for good measure. A three node esxi* cluster connected to a SAN is super basic, super easy to set up and run, pretty low cost and has a large number of individuals and companies who can provide support, maintenance and consultancy who are prepared to work to school budgets. Additionally, I'm not sure one could buy three hosts with redundant power supplies that don't have the CPU grunt to run 60+ single purpose VMs sized to support a school. Thus the 1-vm-per-service is still a perfectly valid design for legacy.

 

 

*other hypervisor platforms are available, and with vmware licencing going the way it has, maybe docker et al's time has come in the on-prem space.

 

But until vendors start shipping containers rather than VMs/interactive installers for their legacy on-prem apps, VMs are going to be the preferred unit of management for school IT teams. We are in the long-tail for this stuff and vendors of the small stuff, if they haven't already got a cloud migration plan, are likely just running down the clock until all their customers swap to the cloud. The market for legacy apps used in schools re-worked to exist happily in containers is (in my rough estimation) insufficient to be profitable.

 

This leaves an interesting situation for the younger techs here - with no vendors shipping on-prem products using the new-shiny container based stuff, are schools a dead-end for those at the beginning of their careers?

 

Honestly maybe I just need someone to show me how its done.

[TechMonkey]

 

This leaves an interesting situation for the younger techs here - with no vendors shipping on-prem products using the new-shiny container based stuff, are schools a dead-end for those at the beginning of their careers?

 

We may need to split this out to a separate thread but, funnily I was thinking about this today as I was looking at an infrastructure refresh and what we will have on site. I think school techs will pivot to network/infrastructure and cloud specialists. We have to have the platform rock solid with everything in cloud so we have to know the network inside and out, especially resilience and internet connectivity. We will have to keep some infrastructure on site, so know what we need (and probably how to cobble it together from 8 year old hardware, some chewing gum and crisp packet!) and how to run minimal setups. & then we will know how to manage and run cloud infrastructure, SSO and administration, as well as the governance behind it all.

 

So, it isn't hopeless or a dead end, we will just have different skills. Or we will continue to run on-site setups and then when the worm turns and everything comes back on site we will be in demand!

[Ditto]

Honestly maybe I just need someone to show me how its done.

Yes!

I mean this in all seriousness. In almost all other job roles, this is called training. Maybe us ITer's do ourselves an injustice by feeding the machine that expects us to continuously re-skill and advance without training.

[dmj]

*other hypervisor platforms are available, and with vmware licencing going the way it has, maybe docker et al's time has come in the on-prem space.

Interestingly, vmware was one of the (fairly) early providers of kubernetes clusters for on-prem, their platform is Tanzu.Google, Rancher et al also provide on premise solutions. anyway, I digress.

Additionally, I'm not sure one could buy three hosts with redundant power supplies that don't have the CPU grunt to run 60+ single purpose VMs sized to support a school. Thus the 1-vm-per-service is still a perfectly valid design for legacy.

I get it in the neck if our servers aren't running at 70% load 95% of the time. There's a certain amount of irony that small schools are able to afford single servers capable of running 60 VM's, sitting idle most of the time - I'd never get away with that now. I fully take the point that small schools aren't going full container based soon, but a large MAT should be maximising their resources.

 

But until vendors start shipping containers rather than VMs/interactive installers for their legacy on-prem apps, VMs are going to be the preferred unit of management for school IT teams. We are in the long-tail for this stuff and vendors of the small stuff, if they haven't already got a cloud migration plan, are likely just running down the clock until all their customers swap to the cloud. The market for legacy apps used in schools re-worked to exist happily in containers is (in my rough estimation) insufficient to be profitable.

Schools software is notoriously bad, but in this case it doesn't usually need to be re-worked to run in a container, just installed into a container. With modern gateways (as opposed to ingress) many would work just fine in containers I've repackaged several things over the years (automated pipelines, not doing it every install!). The most common issue is just poorly designed apps that wouldn't run in High Availability no matter what you tried. Usually they run, but as a "statefulset" in kubernetes so you get some of the benefits, but not all: if a node restarts it has to shutdown the master pod before the replacement is up (in a normal deployment the scheduler will build the replacement and then shutdown the pod) - thus no High Availability and you get x seconds downtime for the pod to reschedule. Plus in the worst case you also get no horizontal autoscaling which means you need to build the pod larger than it would be under little/no load. In these cases it's worth questioning the method if the app is that bad. Unfortunately, the companies that will invest into High Availability apps will prefer to sell the cloud platform as there's higher margins in it with fewer support costs.

This leaves an interesting situation for the younger techs here - with no vendors shipping on-prem products using the new-shiny container based stuff, are schools a dead-end for those at the beginning of their careers?

Honestly maybe I just need someone to show me how its done.

Remember the days when we started running VM's and all the big vendors, Capita et al refused to support virtual machines?It's the same difference, we just installed it anyway, learned how the technology worked and benefited from that. The eager young techs will follow the trends in IT - hopefully, for their sake, not staying too long in schools as I did.