Dazed and Confused!

[jnfarmer]

We've got a weird issue at our place that we cannot get our heads around and I'm reaching out for some help and comfort! Most complaints are around Teams/Zoom video calls, although others mention "general slowness".

 

We've got a section of the building (Head/Deputy Heads/Finance/HR/Bursar/Admissions) that is having constant issues with wired and wireless connections, yet everything we look at and test seems to come back to suggest it is all ok, even though the user experience is not!

 

Here is what we've done so far:

 

Tested all copper cabling, and even though it is at "worst" Cat5E, all tests ok. I'm talking high end fluke test kit, not just flashy lights test.

 

Tested fibre between the switch and our core. Switch is a new Aruba CX 6100, 10G to the core over OM3 8 core fibre. One pair tested not so great in terms of dB loss, but the other 3 pairs were really good, so we moved from "bad" pair to one of the good pairs.

 

We use Lightspeed Filter and WatchGuard firewall. I've gone through those and made sure nothing is blocked/inspected in terms of MS Teams and when I run a connectivity test all comes back ok, except for high jitter, which can then be absolutely fine on the next test.

 

Next thing to try is swap the 10G SFPs back to 1G SFPs to see if that helps in anyway, but we're not seeing any packet loss. And other than replacing the fibre run, which I'm not really wanting to do as nothing suggests that we should, we are at a bit of a loss at to what else to try!

 

Oh, and we use Ruckus One wi-fi which went in last summer and all switch firmware is bang up to date.

 

Help!

[TechMonkey]

Couple of thoughts.

 

Can you replicate the 'issues' and are they tangible? Or just it feels slow? Is it laptops or desktops or both?

If you can replicate, try plugging into the core switch and see if the issue is there and then work back. If the issue is there, then it is probably external and your ISP needs a kicking.

Alternatively, install something like PRTG free and monitor the traffic on ports from a problem user all the way to the core and see what kind of load is being generated.

Presuming switches don't have an logs you can look at to see errors?

[jnfarmer]

I can't. When I go and sit in that area to do some work, including phone calls and Teams video calls all seems fine, which adds to the confusion.

 

I've got SNMP traffic senson on the uplink ports and both ends (core and access switch) and it doesn't look overly loaded.

 

We've had our MSP look at logs on switches and they can't see anything obvious.

[Net-Ctrl]

Hi - It sounds like you have covered a lot of areas. On the Wi-Fi side of things, on the Ruckus One platform does the client troubleshooting page confirm any of the user experience/connectivity issues? Are APs getting sufficient PoE - assuming a possible upgrade last summer may require PoE+ - equally does the switch have the PoE budget to run all APs connected? Have you had a wireless survey completed in the area - or since done has there been any building work that may have changed the environments, or device policy changes, like BYOD/1-to-1.

[pete]

Desktops or laptops? Same hardware as everywhere else (make/model, but also firmware) or different?

 

If you move one of those people to another area for testing (ideally with the same laptop), does it improve or stay the same?

 

Has anything happened that would just affect that area/the people in that area?

 

Do the inhabitants use software that you wouldn't typically have running? Or use it in a different way? Got any memory leaks?

[Davit2005]

WiFi and wireless use the same switches?

 

I'd suggest you monitor the links as @TechMonkey mentions and get more precise timings for the issues happen and what they are. Do you have any services on site that are also effected, if possible can you monitor those too.

 

Intermittent issues are by far the most difficult to troubleshoot :-(

 

@pete makes a good point. Is this jus effecting certain software that only those departments use.

Edited March 28, 2024 by Davit2005

[TechMonkey]

Do the Teams logs (unless MS has put them behind the pay wall) show issues? They are pretty comprehensive and can show issues.

 

One theory, could it be the other party? They have rubbish network/internet and your users are seeing the effect.

[mavhc]

It's unlikely to be your network, but just in case, run iperf3 speed tests on the hour every hour.

 

Make sure they log the exact time the problems occur

 

Make sure everyone's using wired connections to eliminate any radio interference issues

 

Log cpu usage and ram usage

 

Log internet bandwidth

 

Then once you have exact times, go through the logs to find any causes.

[jnfarmer]

So...

 

Our IT estate is a mixture of MacBook/Lenovo/Surface Pros, all laptops and the odd desktop, but all the people in this area of the building are on laptops. Most will be using docks, again mixture of Lenovo/Startech/Surface or a USB-C adaptor with ethernet it, some also just on wi-fi. Some of our offices don't have enough ethernet ports, hence some office based workers using wireless.

 

The troubleshooting on Ruckus isn't showing anything obvious, certainly nothing about under powered APs. Some EAP connection issues, but I've put that down to roaming around the site.

 

We've mentioned that the international calls might be a bit droppy due to the connection the other end, but some "internal" Teams calls are also a bit ropey.

 

We have asked those who do the international Teams calls to try elsewhere in the site, and when they did, it was fine.

[PGridley]

In relation to this one, we've also run numerous Teams Network Assessment Tool reports and compared results in a "known good" area, and in the problem areas.

The switches went in at the same time as the WiFi (or shortly after) and we've had support in stopping broadcast traffic in case it's that. Weird thing is it's still doing it during the Half Term break when you'd expect a lot less traffic as Pupils MacBooks aren't onsite

Edited March 28, 2024 by PGridley

[mavhc]

run iperf to the machines in question then, as well as monitoring cpu/ram

[psydii]

Since you mentioned them, and they don't appear in your log of troubleshooting so far, I think its going to be the docks. Most likely either the usbc cables / usbc ports on the laptops, but it could also be the actual dock themselves.

 

On another tack, have you taken a look at what Wireshark sees when a computer is being slow? Does the traffic look normal or are the 1000's of broadcast packets per second / lots of red and black in the pcap?

[pete]

I'd also examine the docks closely and the quality of the power in that part of the building. I know that doesn't explain the wireless clients though.

 

At one of our sites, USB-C docks work more or less flawlessly in the main buildings.

 

In two satellite buildings (across a public access road and on a different overhead power feed/meter), docks* are unreliable (HDMI flicker/drop, occasional network disconnects) unless fed via a UPS. Powering the docks (via a USB-C brick) may improve it a bit, but won't eliminate the problem.

 

Look at USB power-saving settings too.

 

*all brands.

Edited March 28, 2024 by pete

[sigma]

I had some weirdness at a school where somebody would "momentarily" turn on a printer I didn't know about that had a duplicate IP addy hard coded into it. I was only there for 3 hours a week and it never happened for long enough while I was there for me to find it for ages.

[mavhc]

It's worth turning all static IPs into allocated DHCP IPs just for when devices get reset