Major Network Issues

[CorEnfa]

Good morning!

I'm going to apologize in advance as I have quite a few topics I'm going to cover here and I don't expect any one person to try to solve all of my problems. I'm more here for a sanity check than anything else.

 

I just took over a small to medium sized IT department for a small college. We have roughly 2000 students and up to 900 employees all with multiple devices.

 

My network team has already established the network itself and I see quite a few huge problems here.

 

First, our Wi-Fi is terrible. We're using Aruba APs and WCs with both N and AC WAPs. Here are the issues with wireless:

- Our team has throttled all of the APs to a max of 5mb/s saying that any more than that and the network can't support it (we are tying them back to dated Cisco 10/100 switches then back to a 6509).

- We have TERRIBLE Wi Fi on campus. Despite having 750+ APs on a 95 acre campus. I honestly think some of our APs are stepping on each other.

- Rogue APs are causing major problems. Someone fires up a personal hotspot and everyone in a 50' radius loses complete access or their speed goes to a crawl.

 

 

Second, our VLAN infrastructure seems hosed up.

- I have 180 VLANs. Basically, that's 5+ per building. 1 for standard wireless, 1 for secure wireless, 1 for wired, 1 for the APs, and 1 for printers. Voip also gets its own VLAN in each building too, so in many buildings it's 6 VLANs. I've suggested that all VOIP can go on a single VLAN for all buildings, all wireless can go on its own VLAN or 2 (for address space if needed)

- We need a separation of broadcast domains between faculty/staff and students, but that many seems VERY much overkill.

- I think our outdated switches (2960 and 2940) are having a hard time with that infrastructure and would benefit from a much simpler setup.

 

I welcome your thoughts, suggestions, and overall related stories that might be of help. I'll happily answer questions as they come up.

 

Thanks!!!

J.

[mikeyd101]

hey, sounds like a bit of a nite mare there, we have slightly smaller setup 1500 students 250 staff across 3 main campus + 5 other buildings admin, spec ed, old high school, field house and bus barn.

 

It sounds alot like you have interference on your wifi channels with that many aps, my first thing would be to find out ap locations, number of radios, power and what channels those radios are on. We have no-where near your level only 36 xirrus arrays (222 radios, most are 4 or 8). We have very little performance issues apart from overcrowding 60-70 devices on an array. Our wifi was build for distribution and were migrating toward density.

 

-Are they dual band (2.4 & 5)? or only 5 (n,ac) only, can you change config of radios to help support where needed. if there on 2.4 make sure your only using channels 1,6 & 11

-Are you sure there all interconnects are connected at 100? I had a sneaky video conference machine that had BAD performance, later found out plugged into a 10 only hub.

-Are you doing any network monitoring? I use "the dude" here as its free and does most of what we need monitor switches, machines, ip cameras going down and interlink traffic volume, most snmp and wmi.

- Get some idea of what types of connections/types or devices, few other things to look at would be channel bonding and guard interval (you want short if all your devices support it), and no of radios to use. If you have an overcrowding situation these might help give a bit more performance.

- backbone upgrade, sounds like you need it, have you looked at e-rate (title II)? We have applied this year for a BIG network upgrade and they have a big pool of money on an 80/20, so you only pay 20% of the upgrade cost. Possibly its only a Texas thing.

 

I recently went to a very good session at TCEA this year and they explained how wifi magic happens (or not in some cases). They used a great cheap tool to monitor rf in the air called wi-spy by metageek, I think i've managed to convince the school to get one, it monitors air time which is the only real thing that governs wifi is how much air time is in use. No spare air time = no wifi, and wifi only supports 1 client at a time for each radio. some other tips they gave is to turn down the radio power levels, i'll have a dig through my notes and see if i can find other stuff.

 

We also had a load of extra vlans here, and no network structure. Last year we simplfied vlans and re-addressed everything, it has help alot, seems much faster and none of the weird connection problems we had before.

 

I know there's not an answer in that info, but hopefully this will help get you in the right direction. If you want any specifics drop me a message and i'll give you my email/number.

[CorEnfa]

hey, sounds like a bit of a nite mare there, we have slightly smaller setup 1500 students 250 staff across 3 main campus + 5 other buildings admin, spec ed, old high school, field house and bus barn.

 

It sounds alot like you have interference on your wifi channels with that many aps, my first thing would be to find out ap locations, number of radios, power and what channels those radios are on. We have no-where near your level only 36 xirrus arrays (222 radios, most are 4 or 8). We have very little performance issues apart from overcrowding 60-70 devices on an array. Our wifi was build for distribution and were migrating toward density.

 

 

Mikeyd101 - thanks for the great reply. We too shot for distribution first and density second, but obviously, someone was over-compensating when it was designed originally. ;-)

 

 

-Are they dual band (2.4 & 5)? or only 5 (n,ac) only, can you change config of radios to help support where needed. if there on 2.4 make sure your only using channels 1,6 & 11

-Are you sure there all interconnects are connected at 100? I had a sneaky video conference machine that had BAD performance, later found out plugged into a 10 only hub.

-Are you doing any network monitoring? I use "the dude" here as its free and does most of what we need monitor switches, machines, ip cameras going down and interlink traffic volume, most snmp and wmi.

- Get some idea of what types of connections/types or devices, few other things to look at would be channel bonding and guard interval (you want short if all your devices support it), and no of radios to use. If you have an overcrowding situation these might help give a bit more performance.

- backbone upgrade, sounds like you need it, have you looked at e-rate (title II)? We have applied this year for a BIG network upgrade and they have a big pool of money on an 80/20, so you only pay 20% of the upgrade cost. Possibly its only a Texas thing.

 

- Dual band, 2.4 and 5 with channel restrictions. Though I just found out, we're allowing B and G on the network too. I've asked to have B capability shut off.

- Pretty sure we're all connected at 100.

- We're using Intermapper

- I'm not thinking we're overcrowded for number of devices per AP given our high quantity. Also, we may have 50 devices connected to a single AP, but that factors that a student may have a streaming device, laptop, phone, and tablet, but only 1 or 2 of those are using any considerable bandwidth at a time.

- Our new WAN connection is 10Gb/s into an ASA with 5Gb/s capability (upgradeable to 10Gb/s) to a gig backbone out to 10/100 closet and distro switches. I'm trying to go gig closet to 2 gig distro to 5/10 gig core. That takes serious moola. I'll look into the upgrade stipends. We're in WA, so it's certainly different, but I'll check anyway.

 

I recently went to a very good session at TCEA this year and they explained how wifi magic happens (or not in some cases). They used a great cheap tool to monitor rf in the air called wi-spy by metageek, I think i've managed to convince the school to get one, it monitors air time which is the only real thing that governs wifi is how much air time is in use. No spare air time = no wifi, and wifi only supports 1 client at a time for each radio. some other tips they gave is to turn down the radio power levels, i'll have a dig through my notes and see if i can find other stuff.

 

We're purchasing some tools right now to help evaluate air/radio quality.

 

 

We also had a load of extra vlans here, and no network structure. Last year we simplfied vlans and re-addressed everything, it has help alot, seems much faster and none of the weird connection problems we had before.

 

I'm curious to hear about that some more. Did this help your wireless and wired network? Do you bandwidth throttle any of your clients, APs, or network segments? Did you see more headroom at your switches? Less packet loss, etc.?

 

I know there's not an answer in that info, but hopefully this will help get you in the right direction. If you want any specifics drop me a message and i'll give you my email/number.

 

Your reply was quite helpful. Thank you!

[mikeyd101]

simplifying our vlans along with re-addressing, firstly our original vlans weren't configured correctly so we had alot of routing issues where devices couldn't see other devices and hops between wifi and wired weren't routing correctly. We scraped the whole lot and started again (as well as re-addressing all of our devices) and cut down our vlans to 7, 1 for each campus building, the main place we had more vlans was our high school campus.

 

Re-addressing we changed our mask 255.255.248.0 to 255.255.240.0 to allow for more clients per segment. This helped us alot with device allocation and rules for ip allocation. This may not be the most ideal segmentation but its better than we had and eliminated our routing issues. Mainly having the switches configured correctly our switches that had the most connection vlan issues are hp z5412 / 5406 with 10g interconnects. This is the newest of our switches, most of our network is extreme 10/100 with 1000 interconnections.

 

We have no traffic throttling, most of our wifi arrays are connected with gigabit but not sure they really need it after looking at array throughput and your 5mb/s seems like a good cut off to stop massive peaks.

 

I would definitely look at your vlan configurations/trunking and double/triple check them.

 

Some more food for thought is looking at the switches themselves and seeing how hard there working (cpu).

 

1 of my side projects is to try and cut down on unneeded network protocols ipv6 and legacy protocols, to try and cut any unneeded chatter off the network. Were unlikely to implement ipv6 for quite a while so no need for it on network at this time. I only found out about this a little while ago while reading some other stuff and not likely to make a huge difference, but worth checking, tools like wireshark will help identify.

[CorEnfa]

Thanks again!

 

The switches are long in the tooth and they're hitting pretty close to capacity, so that's on the short list to replace (but that's 60-80 switches. $$$$)

 

The bandwidth throttling is killing us. This is a college campus with most people using wifi. The 5mb/s limitation is way below expectations. But my network manager is telling me he thinks removing restrictions will blow up the network. In my previous network, I had a LOT more clients and no throttling. We had newer switches and a 10g backbone though.

 

Thanks!