Virtualizing Servers

[Nick_Parker]

Hi All

Regarding Virtualizing Servers, which I think is an excellent way to get the most out of hardware as well as safety if a machine should fail, is it a good idea to virtualize servers such as the DC or the exchange server?
I read somewhere that MS doesn't recommend virtualizing DC machines and somebody else mentioned something about servers with DB's on them because of the amount of HDD read/writes?

What do you guys think?

[ssiruuk2]

vmware have got a document on their site testing esx with exchange 2003. Seems to be fine .. They are supporting 2000 users on the test server with no problems. They show comparisons to running it on a physical machine.
Dcs are fine to virtualise. Not that you asked but terminal servers are a big no no - run shockingly as i found out testing some thin clients last year!

[Midget]

so databases, DCs, and TS cant be virtualised, what else is there?

[Ric_]

so databases, DCs, and TS cant be virtualised, what else is there?

Not quite what was said. Terminal Services (or Citrix) should only be virtualised if you have a need for regularly moving or provisioning servers - probably not necessary in a school environment. There is a performance drop - anything from 8%-50% depending on the virtualisation technology used.

There is no need not to virtualise a DC - in fact DCs are one of the best candidates for virtualisation because you can then keep the role separate without piling on all the other innevitable rubbish.

Databases require a bit more care because disk access is an important consideration. If you have a SAN, however, you do not need to put the database within the virtual disk.

[dhicks]

I read somewhere that MS doesn't recommend virtualizing DC machines and somebody else mentioned something about servers with DB's on them because of the amount of HDD read/writes?

Is this because of an issue of some kind with the virtulisation of the server itself, or because of performance issues with the virtualised disk image? I can imagine a heavily-used database might have performance problems doing file access through a file-based virtual disk image (i.e. what the VM sees as a "disk" is actually a file sat on an underlying file system of some kind, making disk seeks and so on act differently from how they should). The obvious solution to this is to use a VM system that can assign a real harddisk to a VM. I know you can do this with Xen by simply sticking a harddrive into a machine and assigning the physical block device in a VM's config file, I'm guessing other systems let you do something similar.

--
David Hicks

[Midget]

i thought there were problems with DCs on VM due to time variences or some such.

I actually want to virtualise if i can so that I can get a seperation of some of my server tasks and utilise more than 4gb RAM

[dhicks]

i thought there were problems with DCs on VM due to time variences or some such.

Hmm, time to ask the All-Knowing Google... Ah, this looks like it'll have the answers:

Considerations when hosting Active Directory domain controller in virtual hosting environments

Most problems would seem to apply to situations where you have multiple domain controllers, all replicating between one another (i.e. taking a snapshot backup of one would confuse the others should you then restore that backup at some later point, etc).


--
David Hicks

[Nick_Parker]

Awesome, thanks for your help guys.

What I'm going to do is a Virtual Machine for the DC/AD/DHCP/DNS
a Virtual Machine for Exchange Server
(FYI - When I was asking about Virtualizing DB's, I was thinking in terms of Exchange's storage being one massive DB)

Thanks for the warning regarding Terminal Servers & VMs, I had thought about setting up a Virtual Machine Terminal Server, I'll put that off for now.

I hear Win2k8's Hyper-V is now RTM so I was thinking about giving that a try...

I have some questions regarding SANS etc, I'm afraid I'm a bit of a n00b when it comes to this side of servers.

I understand the concept of a NAS server, a server with alot of HDDs, shared and connected to the network. You're obvious bottleneck here is the LAN speeds?
What exactly is a SAN/How does it work?
What other options are there?

Thanks!

[Midget]

SAN is basically in its simplest form a scsi or fibreoptic cable attached hard drive array.

[dhicks]

I understand the concept of a NAS server, a server with alot of HDDs, shared and connected to the network. You're obvious bottleneck here is the LAN speeds? What exactly is a SAN/How does it work?

Instead of making file systems available, as a NAS does, a SAN makes block devices available, i.e. disks (or disk partitions, or RAID arrays, depending on what hardware you've set up). Processing machines (i.e. the machines actually running VMs) can access disks over the network. If a processing machine goes down, processing can simply be switched to another machine.

What other options are there?

You're right, LAN speed is an obvious bottleneck, and so is LAN latency, which is something you'll definitely start to notice the more disk operations you have. This is why fibre channel connections are used - not just for the additional throughput, but also for the reduced latency. It's also why SANs are "Storage Area Networks" - the networking infrastructure is dedicated to the SAN, i.e. you have a separate network dedicated to whizzing disk data about the place.

Fibre channel / 10 gigabit ethernet is expensive - enough so to be out of the price range of most schools. Aggregating multiple gigabit ethernet connections together is a realistic alternative (4 gigabits seems to be a kind-of standard - four-way all-in-one network cards are available, and switches that do 4-port aggregation are cheap). Several protocols are available for making block devices available over a network, iSCISI probably being the best known, but you might also want to check out ATA over Ethernet (AoE) - check Wikipedia for a full list of protocols and descriptions of each one.

SANs, in themselves, don't get you any fail-over gains - sure, if a processing machine conks out then another can instantly take over, but if your SAN goes down then everything else stops. Providing for failover thus requires having a replicated SAN, which starts to get really expensive.

Personally, I think a SAN is appropriate for a situation in which your CPU power requirements outweigh your disk I/O requirements, or you're simply running at a large scale, i.e. if you're a medium/large company. In a school situation I'd favour buying large servers with both decent local RAID arrays and fast processors, doing away with all the extra network hardware (and therefore cost) associated with a SAN. I'd still replicate between individual servers (preferably in physically separate locations) for failover purposes, but that could be done with a direct cable between machines, it needn't even involve a switch.

--
David Hicks

[torledo]

Instead of making file systems available, as a NAS does, a SAN makes block devices available, i.e. disks (or disk partitions, or RAID arrays, depending on what hardware you've set up). Processing machines (i.e. the machines actually running VMs) can access disks over the network. If a processing machine goes down, processing can simply be switched to another machine.



You're right, LAN speed is an obvious bottleneck, and so is LAN latency, which is something you'll definitely start to notice the more disk operations you have. This is why fibre channel connections are used - not just for the additional throughput, but also for the reduced latency. It's also why SANs are "Storage Area Networks" - the networking infrastructure is dedicated to the SAN, i.e. you have a separate network dedicated to whizzing disk data about the place.

Fibre channel / 10 gigabit ethernet is expensive - enough so to be out of the price range of most schools. Aggregating multiple gigabit ethernet connections together is a realistic alternative (4 gigabits seems to be a kind-of standard - four-way all-in-one network cards are available, and switches that do 4-port aggregation are cheap). Several protocols are available for making block devices available over a network, iSCISI probably being the best known, but you might also want to check out ATA over Ethernet (AoE) - check Wikipedia for a full list of protocols and descriptions of each one.

SANs, in themselves, don't get you any fail-over gains - sure, if a processing machine conks out then another can instantly take over, but if your SAN goes down then everything else stops. Providing for failover thus requires having a replicated SAN, which starts to get really expensive.

Personally, I think a SAN is appropriate for a situation in which your CPU power requirements outweigh your disk I/O requirements, or you're simply running at a large scale, i.e. if you're a medium/large company. In a school situation I'd favour buying large servers with both decent local RAID arrays and fast processors, doing away with all the extra network hardware (and therefore cost) associated with a SAN. I'd still replicate between individual servers (preferably in physically separate locations) for failover purposes, but that could be done with a direct cable between machines, it needn't even involve a switch.

--
David Hicks

I agree with what david has said, but would like to add that the issue of SAN redundancy can be taken to the nth degree depending on your tolerance for outage and the DR methods in place. A wide area or metro mirror of SANs would provide an even greater degree of availability in case of disaster, but in most cases is overkill. If a school have managed to find the money for a SAN it's unlikely there's enough money available or enough of an incentive for Wide Area Replication. In most cases building redunancy into a single SAN is an adequate step. A second storage array is an option but on a Local level the second array is typically in order to achieve a form of disk based backup or mirroring to secondaty storage. Many storage arrays have adequate capability for mirroring, snapshotting and redundancy in a single box. Insystem. Multipathing of the SAN fabric with dual SAN switches also provides adequate redundancy. Modern storage arrays with additional disk shelves, dual power supplies, battery backed cache and Active/Active disk controllers mean they are very robust. I would never rely purely on the insystem replication and fault tolerance of a single storage array, so while i wouldn't necessarily advocate a backup SAN i'd advocate some form of disk or tape based backup for the primary storage array. This could be direct attached tape, SAN attached tape, or NDMP if the storage array has NAS capability.

Virtualization doesn't mean the tried and trusted rules of backup and restore don't apply...they still do, very much so. What SAN's bring to virtualization is unrivalled flexibility - Virtualization and SANs are inextricably linked becuase much of the cool stuff that a virtualization product is capable of is best achieved using shared storage. And SAN storage allows this shared storage idea to scale beyond 2 servers.

In essense the level of DR you go to with SANs determines the frequency of backup to tape/removable media. For instance some form of in-system replication probably removes the need for daily full or incremental backups but still requires a weekly backup for taking offsite. a disk-to-disk backup system probably removes the need for both daily and weekly backup to tapes but a monthly backup to removable media is probably still advisable due to the requirement to store offsite. A full metro d2d solution, well do you even need tape ? The offsite issues are dealt with, and what is the likelihood of the in-system fault tolerance and replication capabilities of both arrays geographically seperate from conking out on you ?

It's unlikely with a local D2D solution that a monthly offload to removable media is adequate, some weekly backups will still be part of the process, but with a D2d you have more confidence in your backup processes and your less reliant on tape/removable media. You know that there's a good likelihood you can restore from disk, the weekly tape is for the unthinkable catasrophic environental/physical destruction. A monthly tape backup may not cut it as some departments may have updated an awful lot of data between backups that they may not be too keen on replicating, even less so if a weekly tape backup was feasible. Again depends on the end-user base. Will vary from org to org.

[dhicks]

Will vary from org to org.

Which I think sums the whole thing up nicely! I think the advice each of us gives depends on our concept of what constitutes a "school network", which is very much dependant on scale. From my point of view - single private school, probably at pretty much the smallest scale you can get without being part-time at a primary school or similar - things like metro-level networks are way beyond any scale of system we'll be requiring. For us, two large servers would be a major investment, and one active domain controller is perfectly adequate (heck, it isn't even installed yet...).

However, I can imagine a county-wide system in which each school is considered a "branch office" and has a separate domain controller managed by a centralised system, in which case a large data centre with decent redundancy provision becomes necessary. If BSF works properly (!), every state school should be able to quit worrying about messing around sorting out domains, allocating file space, provisioning file servers and ensuring backups, that stuff should just happen (calculating the likelihood of this actually happening is left as an exercise for the interested reader...).

Of course, this being the 21st century and all, we're now ready to crank up the scale another level and have systems utilising thousands of VMs, i.e. skip from having individually managed schools, straight past the stage where we squeeze schools into an organisational structure designed for 1980s mid-sized companies, and go straight on to dynamically-scalable, massively-distributed, multiply-redundant systems built on Google-style commodity hardware. My concern is that many schools are about to spend a stack of time, effort and money in setting up head office / branch office style systems when newer, better models are already starting to be used in production.

There's an interesting audio post over on IT Conversations about Puppet, a programmatic system for setting up and managing large-scale deployments of virtual machines. It's a control system that goes beyond the scale of something like Enomalism, where you click away in a GUI to manage individual machines, instead you define rules about how particular types of VM are to be used. This system is already starting to be used on some large-scale websites - it's the new, up-coming way of utilising the (when you think about it) massive computing resources available to the modern systems designer. It'd be a shame if systems designed for schools were still stuck back in the past.

--
David Hicks

[FN-GM]

One of my schools virtualises all the servers using esx. There should be no reason why you can’t virtualise a DC or exchange server

[torledo]

Which I think sums the whole thing up nicely! I think the advice each of us gives depends on our concept of what constitutes a "school network", which is very much dependant on scale. From my point of view - single private school, probably at pretty much the smallest scale you can get without being part-time at a primary school or similar - things like metro-level networks are way beyond any scale of system we'll be requiring. For us, two large servers would be a major investment, and one active domain controller is perfectly adequate (heck, it isn't even installed yet...).

However, I can imagine a county-wide system in which each school is considered a "branch office" and has a separate domain controller managed by a centralised system, in which case a large data centre with decent redundancy provision becomes necessary. If BSF works properly (!), every state school should be able to quit worrying about messing around sorting out domains, allocating file space, provisioning file servers and ensuring backups, that stuff should just happen (calculating the likelihood of this actually happening is left as an exercise for the interested reader...).

Of course, this being the 21st century and all, we're now ready to crank up the scale another level and have systems utilising thousands of VMs, i.e. skip from having individually managed schools, straight past the stage where we squeeze schools into an organisational structure designed for 1980s mid-sized companies, and go straight on to dynamically-scalable, massively-distributed, multiply-redundant systems built on Google-style commodity hardware. My concern is that many schools are about to spend a stack of time, effort and money in setting up head office / branch office style systems when newer, better models are already starting to be used in production.

There's an interesting audio post over on IT Conversations about Puppet, a programmatic system for setting up and managing large-scale deployments of virtual machines. It's a control system that goes beyond the scale of something like Enomalism, where you click away in a GUI to manage individual machines, instead you define rules about how particular types of VM are to be used. This system is already starting to be used on some large-scale websites - it's the new, up-coming way of utilising the (when you think about it) massive computing resources available to the modern systems designer. It'd be a shame if systems designed for schools were still stuck back in the past.

--
David Hicks

I think the idea of schools as branch offices is about right, we already see it in some LA's but the concept can and will be taken a lot further. I was talking with a guy who worked in the support dept. at a large disk storage company....their UK server and back office presence was being shifted to the US....support staff would be kept on in the UK to manage the servers which would now be remotely managed, you can guess what the next steps are. Do they really need x number of support staff if services are to be consolidated in some super duper air conditioned 24/7 monitored facility in idaho [or wherever] ?

With MANs deisgns now firmly on the path to deploying Ethernet as the defacto standard and the speed boosts that brings, it's going to be less about thinking of each school or site as a branch office but more as outbuildings on a massively extended LAN. Even Ethernet MAN topologies talk about core, distribution and access layers, the same layers LAN administrators have been trying to work from with Local Area Switched Ethernet for years - ethernet has mushroomed beyond it's traditional domain. With the reliability, speed and intelligence now inherant in MANs going forward almost anything can and will be centralized. There's a lot of money to be made from the next-gen intelligent routers which are branch offices in a box...traffic shaping, acceleration, WAFS, WAAS, transparent proxy, packet filter, layer 3 switch...all rolled into each sites WAN access device. Maybe two of them in HSRP config for redundancy, but the box and every switch and module connected to it manageable remotely and with sophisticated O&AMP allowing the centralized support teams in the 'cloud' to detect if the thing burps or falls over.

LA's and MSP's really can take it to this level even for the largest of these 'branch offices', and in the long run they will and that just leaves local onsite staff as toner monkeys. Or maybe through business transformation efficiency they can train and negotiate T&C's for existin staff at a loose end to change toners and do other basic hands on stuff, that is until the remotely controlled robot arm techncian reaches a suitable level of sophistication.

Woops, drifting into BSF stuff there...but i've seen and experienced the future and the future is in the 'cloud'.

Also, David you are right about provisioning file storage....if an LA or MSP were to spend a quarter of a million+ on virtualizing storage and investing in disk based backup they could quite easily manage, provision and scale storage for thousands of users and hundreds of orgs. The technology is already here, and these silos scale to hundred of terabytes and scope to move into petabytes, plenty to provision email inboxes, personal web space and file storage areas for all users. That much responsibility could give some MSP's jitters, and this is the area where they may scale back by mixing and matching local and remote storage centralizing some services and localising others.

[dhicks]

I think the idea of schools as branch offices is about right, we already see it in some LA's but the concept can and will be taken a lot further.

I still think the "branch office" metaphor is somewhat behind the times - as you point out, get some decent all-in-one intelligent managed switches into a school, let them be configured and tuned by centrally-based admins who know about this kind of thing, then let the schools run everything from "the web" - VLE, email, even file access (but have some cunning caching done at the school's side of things, naturally). There's no reason (well, cost, power consumption, er, development...) why this concept couldn't be extended to people's home systems, too - many people have ADSL or cable broadband routers sat at home that are basically small always-on servers, just beef them up a bit until they can cache your school files and so on. It might be a bit premature to start assuming universal home (or even schools) access to MANs as yet.

LA's and MSP's really can take it to this level even for the largest of these 'branch offices', and in the long run they will and that just leaves local onsite staff as toner monkeys.

Or, another point of view, it frees up valuable technical staff's brain cells from doing stuff that can be done automatically and leaves them available to be creating teaching resources and other content. The machines just work, leaving the humans to do the thinking. This, of course, assumes LAs / MSPs think like us and get what kind of system they should be aiming for - I've met some remarkably dim people working for LAs, you know.

--
David Hicks