Whether to virtualise our main servers

[seawolf]

Why would I need a replacement server? Environment would be down as long as it takes HP to get here and fix it

If the worst happens we have a new server onsite next day and Veeam does its stuff.

And in the words of the Headteacher, in the event of a short downtime "we'll cope".

I'll trade the downtime we haven't had in the 6 years we have run this setup for the numerous benefits.

 

Virtualisation may not be the answer to every situation, but you don't understand every situation. I'm not worried.

 

No, I don't understand every situation. However, I do understand what can happen with a single server virtualised environment. It's fine until it all ends in tears.

 

Now, if you say that you have no choice because of financial resources available and you couldn't afford even multiple cheap servers for a physical environment, that is one thing. You're in a tight situation and no choice is very good. Living on the edge is out of necessity.

 

However, if you are saying that what you are doing is good advice for a virtualised environment - then I can state with 100% certainty it is not.

 

If Jollity is in the first set of circumstances then so be it. Risky, but possibly with little choice. If that's not the case, then I want to save Jollity from making a very bad mistake.

[localzuk]

What you have to consider is the size of primary school budgets - they're nearly non-existent for things like servers. Nearly all primary schools I've come across have a single server for everything, with SIMS being on a workstation too.

 

Going from 'one server to rule them all' to 'multiple servers with shared storage' is just not going to be justifiable from a primary IT tech's point of view when their head asks them why they suddenly need to go from 1 to 3 devices etc...

[seawolf]

Crude - Just sat watching Task Manager for a few minutes. Soon get an idea of a servers load capacity. If software (SQL) is hogging all available RAM regardless, go back to the manufacturers recommended spec's, then take an educated guess as to whether or not you think the work load in your environment would need a little more.

 

IMHO, always run SQL server in a VM, and always run it on it's own. That way you can control and restrict it's memory creep and stop it from effecting any other roles.

 

This method is not going to show you your peak CPU, RAM, or network usage on your servers. This is why it is necessary to use a performance monitoring system (perfmon or many of the 3rd party tools available) for a period of time that includes peak usage on your network to determine both the average and peak usage. It is necessary to cater for peak usage in your configurations, not average usage (unless the peak usage is due to a bug or error condition, in which case resolve that first).

[seawolf]

What you have to consider is the size of primary school budgets - they're nearly non-existent for things like servers. Nearly all primary schools I've come across have a single server for everything, with SIMS being on a workstation too.

 

Going from 'one server to rule them all' to 'multiple servers with shared storage' is just not going to be justifiable from a primary IT tech's point of view when their head asks them why they suddenly need to go from 1 to 3 devices etc...

 

That all depends on how you make your case. Solid research, detailed costings and a (business) requirements vs IT expenditures study goes along way. I have been able to triple the IT budget here in the past 5 years. The result - taking a school from a barely functioning IT network that was an incredible burden and detriment on the school's ability to make any use of technology in the classroom - to one that is taking a leading role in the region in the use of IT in the classroom by both teachers and students.

 

And, if you read my comments, I'm actually recommending a small physical environment due to the cost and resource constraints small schools may have. This is a far more reliable and cost-effective solution for a school that can't afford a proper virtualised environment (at least two adequately spec'd hosts with local storage).

[Trev_LCHS]

Really useful and correct information from most folk here, I do suggest you listen to Seawolf as 100% of what he has said is correct.

[tmcd35]

And, if you read my comments, I'm actually recommending a small physical environment due to the cost and resource constraints small schools may have. This is a far more reliable and cost-effective solution for a school that can't afford a proper virtualised environment (at least two adequately spec'd hosts with local storage).

 

Really useful and correct information from most folk here, I do suggest you listen to Seawolf as 100% of what he has said is correct.

 

Not meaning any offense; While the advise boils down to the same - "two well spec'd servers with local storage" - the delivery, IMHO, is a little over complicated given the audience. In an ideal world I'd follow @seawolf's advise to the letter, in reality budgets and timescales tend to call for some more practical decision making.

[sparkeh]

Really useful and correct information from most folk here, I do suggest you listen to Seawolf as 100% of what he has said is correct.

Course yeah, can't fault the techicals and in an ideal world we would all be doing that, but we don't live in an ideal world eh?

I have been able to triple the IT budget here in the past 5 years.

That's awesome and I am happy for you. I take it you live and work in Australia? Things are very different in the UK. Would be interesting to have a show of hands to see how many UK schools have tripled their IT budget recently, I think we would have a lot of empty air.

 

If I had the money to do things the way I wanted, things would be very different. I am just trying to live in the reality of IT in primary schools here.

[tmcd35]

Would be interesting to have a show of hands to see how many UK schools have tripled their IT budget recently, I think we would have a lot of empty air.

 

I ask for triple the amount I'm allocated each year if that counts? - (side note, I actually have a meeting with the SLT after half term to talk about restructuring ICT (read - double the budget); they are currently discussing staff cuts...)

[seawolf]

Course yeah, can't fault the techicals and in an ideal world we would all be doing that, but we don't live in an ideal world eh?

 

That's awesome and I am happy for you. I take it you live and work in Australia? Things are very different in the UK. Would be interesting to have a show of hands to see how many UK schools have tripled their IT budget recently, I think we would have a lot of empty air.

 

If I had the money to do things the way I wanted, things would be very different. I am just trying to live in the reality of IT in primary schools here.

 

Yes, I am in Australia and I certainly understand most schools won't manage to convince management to triple their budget in 5 years. However, it is likely that most IT deparements in most schools will be able to increase their budget if they go about it properly. Doing your homework and then actually asking for more money with good reasons why surprisingly often results in - getting more money. It does little good to do this after the budgets have all been decided for the year, but until they are...

 

If a school is faced with no choice due to budgetary constraints that's one thing, but a risky infrastructure build is still a risky infrastructure build. As I said originally, virtualisation is not always the answer, budget constraints or not.

Edited February 3, 2014 by seawolf

[Trev_LCHS]

if it helps anyone, we have a company here that sells off second hand kit, this kit is pretty good, for example they have some HP dl 380 G5's with 40 Gb of Ram dual processor for less than hundred quid, they have allsorts at different times, very happy to give details. I know its second hand kit, but i have used it for several years and it is reliable, buy it and put it under contract repair, this works for me. I have even fitted some Primary schools for around £600 with a solution that's ok for 3 years. Not right for everyone but an option. Edited February 3, 2014 by Trev_LCHS

[tmcd35]

a risky infrastructure build is still a risky infrastructure build.

 

This is school IT, there's no such thing. We prefer to refer to them as "creative infrastructure builds"

[sparkeh]

Ok last word on this as the OP seems to have needed info.

a risky infrastructure build is still a risky infrastructure build.

We must have a different view on risk.

Our setup has run for 6 years with zero downtime. If the server did have an issue it would be fixed same day or a new server would be run up the next day at the latest. This is acceptable to SMT. We get the benefits of virtualisation at a lower cost and complexity than that of ensuring greater resilience that the managers of the school judge we don't need.

 

It works for us. It will suit some other schools in a similar situation. Running with one host is not always unacceptable.

It wouldn't suit everyone, some people will need as near as dammit 100% guaranteed uptime. Ok then don't follow our model and make your system more resilient.

 

You pays ya money ya takes ya choice

[Trev_LCHS]

I think you all have the same ideas, as has been said you are all talking off the same song sheet. In the risk assessment some can except down time, some cant, but the whole thread was to give Jollie, the best advice he can have, which I think, by the way, has been achieved. Edited February 3, 2014 by Trev_LCHS

[Trev_LCHS]

if your host is part of the domain, some times it has to be, then please follow this advice on preventing a nightmare with system time.

 

We were having timing issues due to the PDC being a Virtual. Hosts look at the PDC for time, but Virtuals take time from the host, so time issues can just spiral. The scenario is, the PDC is a minute out, host looks at PCD changes time to PDC, later, PDC looks at Host and is a minute out so changes time to Host and so on.

 

Solution

 

Configure your PDC as per How to configure an authoritative time server in Windows Server using the “fix it myself”, configuring a PDC to use an external Time Source.

 

Then make sure you go to the Virtual PDC in Hyper V manager, go to setting, and then to integration services, untick time synchronisation.

 

This works well now.

 

Time servers I used when configuring are “1.uk.pool.ntp.uk.0x1 2.uk.pool.ntp.org.0x1” (Don’t forgot to put the ,0x1 or it won’t work).

[seawolf]

if your host is part of the domain, some times it has to be, then please follow this advice on preventing a nightmare with system time.

 

We were having timing issues due to the PDC being a Virtual. Hosts look at the PDC for time, but Virtuals take time from the host, so time issues can just spiral. The scenario is, the PDC is a minute out, host looks at PCD changes time to PDC, later, PDC looks at Host and is a minute out so changes time to Host and so on.

 

Solution

 

Configure your PDC as per How to configure an authoritative time server in Windows Server using the “fix it myself”, configuring a PDC to use an external Time Source.

 

Then make sure you go to the Virtual PDC in Hyper V manager, go to setting, and then to integration services, untick time synchronisation.

 

This works well now.

 

Time servers I used when configuring are “1.uk.pool.ntp.uk.0x1 2.uk.pool.ntp.org.0x1” (Don’t forgot to put the ,0x1 or it won’t work).

 

Yes, good advice. We did the same thing here. Applies to Xen, VMware, Oracle VM, or Hyper-V. Or, you can also use a physical DC as the NTP master.

[Jollity]

I have a few more questions. (Surprise!)

 

Do people tend to use the same backup solution for the host machine system partition as for the virtual machines? If Veeam is for virtual machines only then maybe not. Or is the host machine configuration minimal enough that it can just be recreated? I am guessing that might be true for VMWare, but not for Hyper-V.

 

Similarly do you backup the backup server system partition? (Who backs up the backers up?!)

 

A fellow admin I was talking to yesterday, said that he keeps some roles (including DC) on the host system, and puts others, like Exchange, in Hyper-V virtual machines. My understanding would be that it would better practice to keep every other role off the hosts for security reasons. Is that correct? Are there other reasons for not doing it?

Edit: He also said that one of the virtual servers is a remote desktop server. It would be attractive to have a virtual remote desktop server but I think I read in another thread on here that it was a bad idea. Why?

Edited February 5, 2014 by Jollity

[localzuk]

We run the same backup solution for everything - backup assist. It leverages the backup capabilities built into Windows but puts a better front end and more capabilities at your fingertips.

 

It's usually inadvisable to do anything other than run guest VMs on a hyper-v host, and Microsoft don't recommend it.

 

Regarding backing up the backup system - there should be no need really, as the restoration process should be as simple as "get software used to backup, install on machine, restore from backups to other machines".

 

With Backup Assist there isn't a central backup host anyway. You run it on each host you're backing up. We run it on 3 servers - the 2 hyper-v nodes and our storage server.

[sparkeh]

I wouldn't run anything else on host machine. Our production host (and new replication target) are both off domain boxes with just hyper-v installed.

This afternoon I spun up the server and added hyper-v role in about...20 mins. As a test I copied across a vhd, did some minor config and had the vm up and running in no time. Probably quicker than restoring a backup of the host. This probably goes for backup system as well.

Edited February 5, 2014 by sparkeh

[seawolf]

I wouldn't run anything else on host machine. Our production host (and new replication target) are both off domain boxes with just hyper-v installed.

This afternoon I spun up the server and added hyper-v role in about...20 mins. As a test I copied across a vhd, did some minor config and had the vm up and running in no time. Probably quicker than restoring a backup of the host. This probably goes for backup system as well.

 

Agreed, never run a DC (or anything else really) on a server you also use as a Hyper-V host.

[seawolf]

We run the same backup solution for everything - backup assist. It leverages the backup capabilities built into Windows but puts a better front end and more capabilities at your fingertips.

 

It's usually inadvisable to do anything other than run guest VMs on a hyper-v host, and Microsoft don't recommend it.

 

Regarding backing up the backup system - there should be no need really, as the restoration process should be as simple as "get software used to backup, install on machine, restore from backups to other machines".

 

With Backup Assist there isn't a central backup host anyway. You run it on each host you're backing up. We run it on 3 servers - the 2 hyper-v nodes and our storage server.

 

+1 for BackupAssist. Low cost, flexible, reliable, and easy for even noobies to understand.

 

Veeam is a better backup solution for Hyper-V or vSphere. It is also a lot more expensive and fickle.

[seawolf]

I have a few more questions. (Surprise!)

 

Do people tend to use the same backup solution for the host machine system partition as for the virtual machines? If Veeam is for virtual machines only then maybe not. Or is the host machine configuration minimal enough that it can just be recreated? I am guessing that might be true for VMWare, but not for Hyper-V.

 

Similarly do you backup the backup server system partition? (Who backs up the backers up?!)

 

A fellow admin I was talking to yesterday, said that he keeps some roles (including DC) on the host system, and puts others, like Exchange, in Hyper-V virtual machines. My understanding would be that it would better practice to keep every other role off the hosts for security reasons. Is that correct? Are there other reasons for not doing it?

Edit: He also said that one of the virtual servers is a remote desktop server. It would be attractive to have a virtual remote desktop server but I think I read in another thread on here that it was a bad idea. Why?

 

We have two virtual DCs and one physical DC. The physical DC is recommended in a virtual environment as so many systems are dependent on DNS these days and it also means people can still authenticate if you have issues with your virtual environment. We use a low-spec Dell R210 for the physical DC. It has 4GB RAM, but not much grunt overall.

 

We backup our VMs using Veeam to a FreeNAS server and then have Veeam setup to do a secondary copy to a Drobo B800i connected to the backup server via iSCSI. It is recommended that you always keep at least two copies of your backups. And that you verify them occasionally.

[tmcd35]

What ever you do, if you're using Hyper-V, NEVER run DC on the host. DC has doesn't like multi-homed network cards (pretty much a requirement for Hyper-V). You'll be in for a world of pain if you do.

 

For back up we do the following. Or storage server (based on SMB shares) holds both VHD's and user data (home drives, etc). User data is backed up using Yosemite Backup - pretty much standard, traditional Windows backup ware. The SQL servers (VM's) run backups into backup folder on the storage server which is then backed up again as part of the Yosemite system. This is our daily, incremental, help users out if they loose a file backup.

 

We also have a NAS box I take offsite. The whole shooting match, VHD images and all, is Robocopied on to this once every term. This is out distaster recovery backup.

 

For Terminal Servers - it's in essence a form of virtualisation all ready. Mutliple users shareing one set of CPU/RAM resources. Depending on the number of users they can often require the same kind of spec's as virtualisation host servers to provide a smooth end user experience. So it's considered a bad idea to vitualise because you're adding an extra layer of complexity between the terminal server and the hardware, and you probably wouldn't be using the host for anyother VM's.

 

That said, Server hardware is getting cheap and powerful. I just got 24 cores and 128Gb Ram for around £2.5k. On this kind of host giving over 4 cores and 16Gb Ram to a RDP server is nothing (which is what I plan to do). With powerful hardware, dynamic memory, and easy VHD backups, I think the traditional "don't run Terminal Services in a VM" mentality has passed.

[fiza]

That said, Server hardware is getting cheap and powerful. I just got 24 cores and 128Gb Ram for around £2.5k. On this kind of host giving over 4 cores and 16Gb Ram to a RDP server is nothing (which is what I plan to do). With powerful hardware, dynamic memory, and easy VHD backups, I think the traditional "don't run Terminal Services in a VM" mentality has passed.

 

Can I ask what server it was and where you got it?

Also I read somewhere that you couldn't run Hyperv in a VM as its already running on the host? Is that not right anymore?

[robjduk]

Hyper-V should be easy enough to set up again should it fail.

[tmcd35]

Can I ask what server it was and where you got it?

Won't go in to specifics as the CPU(s) was end of line and I badgered heavily on price But it's from @VeryPC. Novtech have managed similar for me in the past. Both good bespoke manufacturers who know how to treat their customers.

 

Also I read somewhere that you couldn't run Hyperv in a VM as its already running on the host? Is that not right anymore?

As far as I am aware that's still true. Sounds like a bonkers thing to try IMHO. Bear in mind I was talking about running Terminal Server in a VM which doesn't rely on Hyper-V.

Edited February 6, 2014 by tmcd35