VLANs in Virtualisation

[garethEds]

Evening all,

 

We are now in possession of our two servers ready for virtualising our older pieces of hardware. My funding was cut in half during the project and the funds I had for a SAN are now gone. Gutted - but there is nothing I can do at the moment.

 

I am moving forward with virtualising some of the smaller servers and they will be stored on the servers (which have large hard drives mirrored).

 

I've read somewhere that you should keep your host servers in a seperate VLAN with a different IP range yet can keep your virtualised servers on them with the 'main' ip range. Can anyone advise me why this is so?

 

Currently, all servers are on 10.180.8.xxx/255.255.248.0 - I want to keep this for them when they are virtualised. Can I put my two host servers on something different that doesn't confilict wich anything the LEA has in place (for example 192.168.0.xxx) - why would I do this? Have I read it right?

 

Many thanks

 

Gareth

[geezersoft]

How many NICS are there in each server and what hypervisor are you going with?

[Dave_O]

Evening all,

 

We are now in possession of our two servers ready for virtualising our older pieces of hardware. My funding was cut in half during the project and the funds I had for a SAN are now gone. Gutted - but there is nothing I can do at the moment.

 

I am moving forward with virtualising some of the smaller servers and they will be stored on the servers (which have large hard drives mirrored).

 

I've read somewhere that you should keep your host servers in a seperate VLAN with a different IP range yet can keep your virtualised servers on them with the 'main' ip range. Can anyone advise me why this is so?

 

Currently, all servers are on 10.180.8.xxx/255.255.248.0 - I want to keep this for them when they are virtualised. Can I put my two host servers on something different that doesn't confilict wich anything the LEA has in place (for example 192.168.0.xxx) - why would I do this? Have I read it right?

 

Many thanks

 

Gareth

It's for security reason. So Tommy User can't go scanning around your curriculum ip range and chance upon your VI and also so V motion and management traffic stay separate from the normal network traffic. I know one or two implementations that use the same range and have never had any problems. These are schools were talking about, not high security implementations. Up to you, if you're anal about security then use a different ip range (you don't actually need a separate vlan for that to be fair). If is fear about the amount of traffic then with two hosts in this environment as long as you have Gb connectivity, you won't have a problem

Edited March 30, 2013 by TheScarfedOne

[ConradJones]

The only reason not to VLAN it is if you can't. ie. the hardware doesn't support it.

[Dave_O]

The only reason not to VLAN it is if you can't. ie. the hardware doesn't support it.

 

Blimey! It would have to be really old kit not to support vlans. I remember back in the 80s we had non-layer 2 device. Not seen one since the though.

[ConradJones]

It would have to be really old kit not to support vlans.

or cheap

[Dave_O]

or cheap

 

I'm not sure anyone actually sells non-layer 2 switches when you can pick an 8 port L2 for less than £50 and if you're talking cheaper than that, then you're in the wrong job.

[ConradJones]

I'm not in the wrong job mate. Plenty of people who are though. I wouldn't entertain buying something that cheap, plenty of people who would though as they can't see past the next 5 minutes. Edited March 30, 2013 by ConradJones

[localzuk]

To the OP - as mentioned, its a security issue. Putting them in their own VLAN, which is segregated from the rest of the network isolates them from prying eyes. However, in most school setups, I'd say its a bit overkill - the infrastructure doesn't face that sort of threat in a school, or at least it shouldn't, due to all the other security precautions in place (locked down clients etc...).

 

However, there is another issue with giving it different IPs etc... And that is one of availability. In an ideal world, you would have a separate management interface to your client interface (the one the guest machines use). This way, if the client interface is 100% utilised for whatever reason, you can still connect and manage the machines. Whether this is on a separate VLAN or not is up to you.

 

In a further ideal world, you'd have 3 interfaces per box, with your SAN in place too. 1 would be the management interface, 1 the client, and 1 for iSCSI (which according to best practice, should be on a separate switch entirely). That way, everything is split up, nothing can overload each other etc... However, you'd then want to go to 6 such interfaces, (on at least 2 different controllers, 3 on each), to allow for failure...

 

The numbers could increase and increase though, as you might also want 2 switches to add redundancy etc...

[glennda]

Basicly what @localzuk said - although to be honest I recommend a lot more than 1 interface for client traffic. I tend to have 2 x ISCSI, 2 x management 2 x VMotion and 4 x client traffic split across the main board and a PCI cards.

[localzuk]

Basicly what @localzuk said - although to be honest I recommend a lot more than 1 interface for client traffic. I tend to have 2 x ISCSI, 2 x management and VMotion and 4 x client traffic split across the main board and a PCI card.

 

It all depends on cost and what sort of interface though. We have a single interface for traffic on our servers, but it is 10Gbps so works great.

[glennda]

It all depends on cost and what sort of interface though. We have a single interface for traffic on our servers, but it is 10Gbps so works great.

 

Yup but even then you ideally need 2 for redundancy

[ConradJones]

It all depends on cost and what sort of interface though. We have a single interface for traffic on our servers, but it is 10Gbps so works great.

 

dual 10gbe here. management / vmotion is still on the 1gb for no other reason than we haven't moved it yet. will get removed this easter.

[localzuk]

Yup but even then you ideally need 2 for redundancy

 

Indeed, but ideally you'd also need 2 switches for them to plug into as well... So basically, you're just drawing a line at a different point with your cost-benefit analysis.

[glennda]

Indeed, but ideally you'd also need 2 switches for them to plug into as well... So basically, you're just drawing a line at a different point with your cost-benefit analysis.

 

oh yes completly i'm going with an ideal scenario and also dual links between those switches.

[ConradJones]

this is how ours is setup

[garethEds]

How many NICS are there in each server and what hypervisor are you going with?

 

Hi @geezersoft

 

There are 6 NIC ports in each server with + 1 management port. 4 + 1 as standard in the server and then an extra network card.

 

We will be using HyperV2012 but hosted servers will only be 2008R2 as the LEA do not support 2012 at the moment.

 

Gareth

Edited March 30, 2013 by garethedmondson
Didn't answer all the questions.

[garethEds]

The only reason not to VLAN it is if you can't. ie. the hardware doesn't support it.

 

Our switches can support VLANs.

 

Gareth

[Dave_O]

dual 10gbe here. management / vmotion is still on the 1gb for no other reason than we haven't moved it yet. will get removed this easter.

 

I would be interested to see what the utilisation (percentage wise) of the 10Gb connection is during a normal school day. We have a 4Gb trunked connection from each ESX and have never seen any of the nics that make up the trunks go much above 30%. This is across 3 ESX servers hosting 48 servers including SIMS, Exchange, Sharepoint, file services etc. The basic question is then what is the justification for 10Gb connections in a school environment? I seem to remember having this conversation before and never really getting a satisfactory answer.

[SYNACK]

I would be interested to see what the utilisation (percentage wise) of the 10Gb connection is during a normal school day. We have a 4Gb trunked connection from each ESX and have never seen any of the nics that make up the trunks go much above 30%. This is across 3 ESX servers hosting 48 servers including SIMS, Exchange, Sharepoint, file services etc. The basic question is then what is the justification for 10Gb connections in a school environment? I seem to remember having this conversation before and never really getting a satisfactory answer.

 

30% usage does not nessisarily mean that that is all it would use and is an over spec, it could easily indicate a bottleneck in storage speed, virtual host CPU or memory queues or contention on the rest of the network fabric. Sure it could mean that in your situation it is all that you need but without all the variables we can't know if this is right even for your site let alone others.

 

48 servers, what the heck are you running there?? How did you end up with so many?

[Dave_O]

Our switches can support VLANs.

 

Gareth

 

Thanks for clarifying that one!

 

As you can see from the attached image I use two IP ranges 10.blah.131.? for management & vmotion and 10.blah.16.? for standard curriculum traffic (subnets withheld for legal reasons:)). There is another VLAN (5) which uses the curriculum nics which is for DMZ stuff. This has another IP range 192.blah.blah.? Any layer 2 switch can deal with this.. Hope this helps

[ConradJones]

30% usage does not nessisarily mean that that is all it would use and is an over spec, it could easily indicate a bottleneck in storage speed, virtual host CPU or memory queues or contention on the rest of the network fabric. Sure it could mean that in your situation it is all that you need but without all the variables we can't know if this is right even for your site let alone others.

 

+ Even if you have no bottle necks else where just because thats all you are using today doesn't mean its all you will use in a year or two or three and as the last set of switches stuck around for 8 years, i certainly don't want to be running on a 1gb backbone in 7.5 years time.

[Dave_O]

30% usage does not necessarily mean that that is all it would use and is an over spec, it could easily indicate a bottleneck in storage speed, virtual host CPU or memory queues or contention on the rest of the network fabric. Sure it could mean that in your situation it is all that you need but without all the variables we can't know if this is right even for your site let alone others.

 

48 servers, what the heck are you running there?? How did you end up with so many?

 

1. That is true, this figure is an average over a period of time (1 half term's figures in fact) taken on a daily basis between 8:80am and 3pm (normal school day) and represents that average for that day (yes there were some spikes each day but none maxed out any of the nics). This figure was then recorded every school day for the half term and averaged over that time period.

 

As for bottlenecks and more information related to possible variables that may affect the setup:-

 

See post #42 on this thread

 

http://www.edugeek.net/forums/thin-client-virtual-machines/68166-my-conclusions-vdi-other-things-3.html

 

for details of ESX specs. All have 100Gb of memory and are at about 80% usage. CPU usage monitoring (again over a protracted period of time) shows a maximum of 15% CPU usage (Exchange & SIMS).

 

 

Storage is IBM V7000 (46 x 600Gb 10K SAS, 2 x 200Gb SSDs) with 8Gb fibre SAN switches and HBAs. Switches are HP 3500yl aggregators with 5406zl at core and 4200vl at the edge (ie Gb to desktop) all with a minimum 2Gb trunked fibre between each.

 

 

2. As to the 48 servers... all sorts of stuff read post #6 on this thread.

 

http://www.edugeek.net/forums/thin-client-virtual-machines/84498-virtualisation-other-stories.html

 

 

I have collected a vast number of VI performance metrics over the past 6 years and am happy to share specific tests and how they were collected. What I have not seen is any figures that justify (in a school environment) the use of 10Gb fibre. As to the "future proofing" over the next 4-5 years or so, believe me schools requirements will not change that much in the short term, in fact their local bandwidth needs will, more than likely, change downwards as potentially more services move off site.

[localzuk]

No-one has mentioned 10GbE fibre. My host servers have 10GbE ports onboard, and the storage server has it as an extra card. They plug into an 8 port 10GbE module in our HP 5406zl switch, which was full. So, our purchasing decision was to buy the 1GbE versions of the servers and then trunk things, or just spend the £2k on a 10GbE module for the core, and then we used copper 10GbE cables (gbics formed into the cable).

 

Worked out cheaper for us than going 1GbE.

[SYNACK]

1. That is true, this figure is an average over a period of time (1 half term's figures in fact) taken on a daily basis between 8:80am and 3pm (normal school day) and represents that average for that day (yes there were some spikes each day but none maxed out any of the nics). This figure was then recorded every school day for the half term and averaged over that time period.

 

As for bottlenecks and more information related to possible variables that may affect the setup:-

 

See post #42 on this thread

 

http://www.edugeek.net/forums/thin-client-virtual-machines/68166-my-conclusions-vdi-other-things-3.html

 

for details of ESX specs. All have 100Gb of memory and are at about 80% usage. CPU usage monitoring (again over a protracted period of time) shows a maximum of 15% CPU usage (Exchange & SIMS).

 

 

Storage is IBM V7000 (46 x 600Gb 10K SAS, 2 x 200Gb SSDs) with 8Gb fibre SAN switches and HBAs. Switches are HP 3500yl aggregators with 5406zl at core and 4200vl at the edge (ie Gb to desktop) all with a minimum 2Gb trunked fibre between each.

 

 

2. As to the 48 servers... all sorts of stuff read post #6 on this thread.

 

http://www.edugeek.net/forums/thin-client-virtual-machines/84498-virtualisation-other-stories.html

 

 

I have collected a vast number of VI performance metrics over the past 6 years and am happy to share specific tests and how they were collected. What I have not seen is any figures that justify (in a school environment) the use of 10Gb fibre. As to the "future proofing" over the next 4-5 years or so, believe me schools requirements will not change that much in the short term, in fact their local bandwidth needs will, more than likely, change downwards as potentially more services move off site.

 

1. again, in your school. Push some heavy 3D or movie files over that and your calcs go out the window

 

2. OMG you have taken seporation to the next level in that, I know that memory dedupe exists but that kind of seporation is very specialised. I hate to think of the queues your VMs have to wait in to get access to the network cards propper and all the traffic on virtual DMA mapping.

 

Impressive storage but 8GB shared between that many hosts still provides a bottleneck. We also have a hp 5412 which is a nice bit of equipment but again, every school and implementation is different.