Identifying bottlenecks

[simpsonj]

Not sure where to place this question as it could be any number of things, but wired networks seemed the best fit.

 

Something is causing a bottleneck on my network that restricts bandwidth to around 150mbs:

 

The internet line is 800mbs and can absolutely deliver that speed.

The network backbone is, at minimum, 1Gbs fibre

The core switch is newly installed over the summer and can handle upto 10Gbs throughput

Wireless traffic is vLanned off

 

However, once the students turn on their Chromebooks (around 700 of them) the network's internal and external traffic slows to an absolute crawl.

 

Looking at the Wireless management in Meraki, when Wireless network traffic is around 0-110mbs, everything is fine, when it reaches 120mbs+, this is when the problems occur. I could understand the problems if our wireless network was reaching 500-600mbs, but surely the network capacity is 800mbs, so I shouldn't be having the issues we're having at such low levels?

 

I've removed as many switches as possible, and we only have one IT Suite, so the vast majority of our traffic is wireless. Inter Server speed is around 500-800mbs. When monitoring the Core switch ports, the capacity of any of the uplinks never exceeds 20%.

 

I've been trying to resolve this issue for months, and so far the only method that has worked is to limit the bandwidth of the Chromebooks to a low level so it doesn't saturate the network. But I feel that I'm not getting the most of this network!

 

Any ideas on either what could cause a bottleneck and/or how to identify a bottleneck on a network?

[RobD]

Are most of your wireless clients connecting on 5Ghz or 2.4Ghz? Also what channel width are you using, your clients could be fighting over air time if its set to 80MHz or 160MHZ and you have a lot of clients connected to an AP.

[simpsonj]

All devices are 802.11ac minimum, so connecting on both channels.

It also impacts on wired devices as well, which shouldn't be affected by wireless channel congestion...

[psydii]

Your wired speeds are reduced to 150Mb/s when the Chromebook are all online?

 

 

If so, stick Wireshark on the lan and see what is happening. My guess would be some broadcast/multicast traffic flooding.

 

Meraki APs can also capture wire and wireless-side traffic as pcap files which can be imported into Wireshark for review.

[FN-GM]

Is it a case of your internet connection being completely saturated? 700 Chromebooks (plus other bits and pieces) sharing an 800 meg connection doesn’t leave much breathing space.

 

I am not sure form the OP of the internet connection is monitored and how utilised it is.

Edited September 8, 2022 by FN-GM

[simpsonj]

Is it a case of your internet connection being completely saturated? 700 Chromebooks (plus other bits and pieces) sharing an 800 meg connection doesn’t leave much breathing space.

 

I am not sure form the OP of the internet connection is monitored and how utilised it is.

 

Sorry, should have added, looking at the traffic on the Smoothwall, internet traffic is around 100-200mbs, it doesn't hit peak bandwidth (800mbs) when network issues arise.

 

- - - Updated - - -

 

Your wired speeds are reduced to 150Mb/s when the Chromebook are all online?

 

 

If so, stick Wireshark on the lan and see what is happening. My guess would be some broadcast/multicast traffic flooding.

 

Meraki APs can also capture wire and wireless-side traffic as pcap files which can be imported into Wireshark for review.

 

I suspect you're right, I'm just not great at interpreting Wireshark results. I guess it's time to 'get gud'.

[simpsonj]

Your wired speeds are reduced to 150Mb/s when the Chromebook are all online?

/QUOTE]

 

Sorry, I should have been clearer, our wired speeds are crippled when the Chromebooks are online, pinging 8.8.8.8, Smoothwall or the core switch either times out or is 100-200ms

[psydii]

Sorry, should have added, looking at the traffic on the Smoothwall, internet traffic is around 100-200mbs, it doesn't hit peak bandwidth (800mbs) when network issues arise.

 

- - - Updated - - -

 

 

 

I suspect you're right, I'm just not great at interpreting Wireshark results. I guess it's time to 'get gud'.

 

When you are seeing the sorts of problem you have - I would expect that the problem will jump out at you.

 

To 'get gud' just take a capture on an isolated switch with only a couple of hosts to see a baseline for normal.

Then compare it to a capture from your live network - the anomalous traffic will likely be *REALY* obvious.

After that google is your friend.

 

(its like differentiating a riot on Oxford Street from normal busy weekend shopping day)

[simpsonj]

Thanks, will take the plunge this afternoon!

[mavhc]

So you can get max speed when the chromebooks aren't online?

 

Are they doing p2p updates? Is there a new update they're all installing and sharing between each other? https://support.google.com/chrome/a/answer/3168106?hl=en#zippy=%2Ccache-updates-to-reduce-bandwidth

[FN-GM]

 

Sorry, I should have been clearer, our wired speeds are crippled when the Chromebooks are online,

 

Is it just the wired? Do you happen to have a network diagram to help please?

[slugshead]

Recommendations for switch capacity in schools on the core / backbone school IT network are:

 

10Gbps for Secondary and larger primary schools utilising lots of digital learning

1Gbps for smaller and rural primary schools.

 

sauce: https://hwb.gov.wales/support-centre/education-digital-standards/router-and-switches-standards/

 

I know this applies to Wales, but there's some damn great guidance there. Time to spend some money...

[simpsonj]

So you can get max speed when the chromebooks aren't online?

 

Are they doing p2p updates? Is there a new update they're all installing and sharing between each other? https://support.google.com/chrome/a/answer/3168106?hl=en#zippy=%2Ccache-updates-to-reduce-bandwidth

 

That would explain a lot... Thanks!

 

There's a mode on Meraki (NAT mode) that might solve that particular issue.

[simpsonj]

Recommendations for switch capacity in schools on the core / backbone school IT network are:

 

10Gbps for Secondary and larger primary schools utilising lots of digital learning

1Gbps for smaller and rural primary schools.

 

source: https://hwb.gov.wales/support-centre/education-digital-standards/router-and-switches-standards/

 

I know this applies to Wales, but there's some damn great guidance there. Time to spend some money...

 

I don't disagree, but if I can't maximise a 1Gb network, 10Gb isn't going to help!

[supportman]

Maybe try and get a few windows laptops and simulate a wireless capacity fill. Then check if a similar thing happens.

 

Then you can issolate wether its your wireless, chromebooks or some kind of network issue.

[simpsonj]

Is it just the wired? Do you happen to have a network diagram to help please?

 

Not, not just the wired, everything runs slow.

 

An updated Network Diagram is in the works, but essentially we have one core switch and 14 edge switches connected via Fibre with 52 Access Points all connected at 1Gb.

[slugshead]

I get that it would be nice to get 100% out of 1gb links, but with those kind of numbers of devices, you're not going to have a good time. I would at least if 10Gb is not possible, bond some 1Gb links and leverage multipath stuff.

 

Here's some things I've done in the past when I had a 1000 devices in a BYOD environment and a 100Mb internet connection (Which was shared with the corp vlan). Slightly different but could be of some use.

 

* Block updates during core hours - Smoothwall has a category for this to make it easier. You can also do this with chrome policies.

* Apple caching server

* Allow BITS to traverse to the wired network and host a dedicated machine for it.

* Block "bandwidth wasting activities*

* Host an Adobe update repo on site.

* Change bandwith allocation, think users/groups/vlans and percentages that can be allocated.

* QoS - This is done at layer 3, assuming you have the capabilities for that?

* Block windows updates from internet apart from your WSUS server.

[FN-GM]

Not, not just the wired, everything runs slow.

 

An updated Network Diagram is in the works, but essentially we have one core switch and 14 edge switches connected via Fibre with 52 Access Points all connected at 1Gb.

 

What switches are you running? If something like Cisco what SDM template are you running?

[simpsonj]

All HP Switches

The only config on them is the vLans and spanning tree enabled (so no QoS defined)

[tosh74]

check spanning tree changes on your switches (make sure your core switches have a lower Switch priority than your edge switches)

 

show spanning-tree

 

check for Topology change count and Time Since Last Change . Your CST Root MAC Address should be your core switch and IST Regional Root MAC Address should be the local switch.

[RedwayNetworks_Marketing]

Hi Simpson,

 

It's a difficult one to diagnose without more info however, large numbers of wireless clients in the same subnet can cause problems if there are lots of broadcasts as these generally go to all clients.

Making sure minimum data-rates are set appropriately so slower devices don’t drag the network down.

Making sure that non-overlapping channels are used.

Are access points appropriately specked for the number of clients connecting?

Check spanning tree (if used) to make sure the core switch is appropriately elected as root bridge to prevent obscure, underpowered edge switches becoming root and then causing sub-optimal pathing on the switched side.

Make sure there are no speed/duplex mismatches.

 

Hope this helps

[EduBadger]

Try monitoring your smoothwall in the live web filter area, it would not surprise me if its blocking stuff and causing you issues that make it look like a bottleneck. Call smoothwall and explain the issue they are pretty helpful.

[timbo343]

@simpsonj

 

what version of release are you on with your smoothwall? We are on 61 and we are having some issues with our box which we are struggling to get to the bottom of. The webfilter / proxy / guardian just falls over.

[simpsonj]

Hi Simpson,

 

It's a difficult one to diagnose without more info however, large numbers of wireless clients in the same subnet can cause problems if there are lots of broadcasts as these generally go to all clients.

Making sure minimum data-rates are set appropriately so slower devices don’t drag the network down.

Making sure that non-overlapping channels are used.

Are access points appropriately specked for the number of clients connecting?

Check spanning tree (if used) to make sure the core switch is appropriately elected as root bridge to prevent obscure, underpowered edge switches becoming root and then causing sub-optimal pathing on the switched side.

Make sure there are no speed/duplex mismatches.

 

Hope this helps

 

Thanks for the tips.

 

For anyone else wondering how you make your core switch the root bridge for spanning tree, I found this guide very helpful (for HP Switches)

 

One of those things I felt I should have picked up over the years...

 

I'm going to remove my last 100Mbs switch over the next few days to see if that helps, it's connected at 1GB, so shouldn't slow the network, but everything helps.

[simpsonj]

@simpsonj

 

what version of release are you on with your smoothwall? We are on 61 and we are having some issues with our box which we are struggling to get to the bottom of. The webfilter / proxy / guardian just falls over.

 

On release 60, so maybe I'll hold off on that update!

 

Smoothwall may or may not be the culprit, I'm getting a new and improved box over the next few months, so I'm hoping that will confirm one way or the other.