Tracing network/internet issues

[mdrabble]

Not sure where to put this......

 

Past 2 years have been issue free across the network infrastructure and on the whole staff have been very happy with network performance.

 

Since September staff have been complaining about slowness in general and also at times internet issues.

 

Nothing has change on the network infrastructure side of things, only real change has been moving from Windows 10 1803 to 1909.

 

Lastnight when running schoolcloud Parents Evening, all staff all reported issues ranging from random disconnects to audio issues. - Devices were a mixture of wired desktops to wireless devices.

 

We are running a 300MB connection (soon to be upgraded to 1GB) - smoothwall traffic graphs showed all ok, and LibreNMS graphs also showed everything as ok - when I looked at the smoothwall graphs and the core switch port used to connect smoothwall to the LAN, both we showing approx 50mb of traffic usage.

 

I've asked our ISP to check the line incase of issues.

 

Anyone any other suggestions what I can look at to try and diagnose these intermittent issues?

 

Will try and attach a network diagram soon as I can install Visio to open it!

 

Cheers

[FN-GM]

Slow DNS resolution can give the illusion of slow network connections. Have you had a look at this? Maybe look at your internal dns and then your dns forwarders for public domain names.

[mdrabble]

Internal DNS use Domain Controllers which then forward to Smoothwall which then forwards to GoogleDNS

[HC_Netman]

 

Lastnight when running schoolcloud Parents Evening, all staff all reported issues ranging from random disconnects to audio issues. - Devices were a mixture of wired desktops to wireless devices.

 

 

We ran a parents evening on schoolcloud last night as well and had similar reports of disconnects so I wonder if there could have been an issue their end last night? We have run parents evenings earlier this year with no issues.

 

Sorry I know this doesn't help answer your actual question but thought it was worth mentioning in case you were investigating an issue that could easily be someone else's problem!

[mavhc]

First step is I'd just open 4 cmd windows and have 4 ping -t commands running

To, say, the switch, the server, the gateway, and google.com

 

See if there's any blips, timeouts, etc. Ctrl-break will show current stats and continue running the ping.

[mdrabble]

Pings to smoothwall, fibre core and copper core switches are all around <1ms with the odd ping being slightly higher.

 

Pings to google all approx 15ms

 

No timeouts

[FN-GM]

Internal DNS use Domain Controllers which then forward to Smoothwall which then forwards to GoogleDNS

 

Is there any reason your domain controllers don’t forward directly to Google?

[mavhc]

Pings to smoothwall, fibre core and copper core switches are all around <1ms with the odd ping being slightly higher.

 

Pings to google all approx 15ms

 

No timeouts

 

Leave them open all day for long term testing.

 

https://www.senki.org/network-operations-scaling/dns-latency-and-performance-test-tools/

 

What are you using where the audio drops?

 

run a few continuous "meetings" with music playing or something to check the audio, with the software you use, and other software

Edited December 4, 2020 by mavhc

[mdrabble]

Smoothwall is also our firewall and in a previous post I asked about DNS settings I had this reply

 

When Smoothwall is the firewall, we normally setup DNS like you describe but it would be a good idea to also add the ISP DNS servers to the forwarders and set the internal AD DNS servers to use Smoothwall as their forwarder. This shortens DNS path for the entire network instead of asking the internal DNS servers to use root hinnts or ISP DNS as forwarders.

[mdrabble]

What are you using where the audio drops?

 

Parents Evening on SchoolCloud using Webcams - audio normally fine on teams calls (or rather I haven't had any complaints about audio until now)

[mavhc]

Probably their end, everyone's using them at the same time.

 

You'll probably need to be able to say that at the exact same time Teams calls were working fine though before they'll stop blaming your internet connection.

[psydii]

Worth checking your wifi AP firmware. We had problems this September and October (during the first lockdown most laptops got upgraded to 1909 / 2004 from 1803/1809). All glitching stopped after the firmware was deployed.

Prior to the firmware update, we found a few switches that were unhealthy (check for errors on ports and spanning tree recalculations), sorting these also helped a lot.

[howartp]

https://www.senki.org/network-operations-scaling/dns-latency-and-performance-test-tools/

 

Smoothwall is also our firewall and in a previous post I asked about DNS settings I had this reply

 

Aha, thanks for both these posts.

 

We have Internal > Smoothwall > ISP as well, and it's the Exa (our ISP) DNS servers we have been using. (82.219.4.24 + .4.25)

 

Using the GRC DNS Benchmark tool on the link above, one of the Exa servers is returning invalid replies and the other is not responding consistently!

 

Now changed to OpenDNS which is consistently high in the benchmark test.

[Davit2005]

Is there any reason your domain controllers don’t forward directly to Google?

 

This ^^

 

Ideally You want reliable DNS servers as forwarders on your DCs.

 

So either Google or CloudFlare or Quad9 or others. I never point DC DNS to firewall, router or anything but external DNS. Personally, I have my firewalls pointed back to my DCs. Not sure why Smoothwall needs this but I have no experience of using them maybe they can advise why but suggest you ask a the techinical question before you consider changing.

 

Maybe that your internet provider restricts DNS queries to specific IPs??

Edited December 4, 2020 by Davit2005

[Jcx500]

This ^^

 

Ideally You want reliable DNS servers as forwarders on your DCs.

 

So either Google or CloudFlare or Quad9 or others. I never point DC DNS to firewall, router or anything but external DNS. Personally, I have my firewalls pointed back to my DCs. Not sure why Smoothwall needs this but I have no experience of using them maybe they can advise why but suggest you ask a the techinical question before you consider changing.

 

Maybe that your internet provider restricts DNS queries to specific IPs??

 

We do this as well ours point to our isp dns and we have Google’s as a backup

[mdrabble]

Thanks all, will put our internal DNS forwarders to Google and our ISP and see how that goes.

[mdrabble]

Love how our staff have been saying it must be a bandwidth issue - they all must be experts now!!

 

Anyway ISP have confirmed our usage and it tallies with LibreNMS graphs so when they were using the parents evening system, it using a fraction of our limit so rules out bandwidth issues.

[ticktock]

We had a school cloud parent's evening last night and had sound and video issues as well. They lasted for around half an hour then everything was ok again. Our first parent's evening was a disaster with school cloud (a few weeks back) with something failing at their end which they admitted to.

[FN-GM]

Not sure why Smoothwall needs this but I have no experience of using them maybe they can advise why but suggest you ask a the techinical question before you consider changing.

 

I used Smoothwall for many years and never had it configured that way. Unless something has changed recently. Maybe @tom_newton can clarify.

[RobD]

I had similar issues at a site a while back. Slow Tuesday they were calling it.

 

I tested internally between clients and servers with iperf (or jperf) to rule out switches,

Checked the servers performance via the hypervisor and tested the local servers disks with CrystalDiskMark to rule out the servers,

Tested the wireless between AP and client with the android app Cloudcheck,

Checked the bandwidth which didn't seem constrained.

 

All seemed fine so I moved onto the client, cut a long story short it was the client hardware but I had to prove it.

[smarties11]

We are a SmoothWall School, and ever since we deployed it several years ago our DNS path has always been SmoothWall -> DCs -> ISP with no issues. I'd suggest this route.

 

EDIT: just to clarify for clients its client -> DCs -> ISP. Above was meant for what is set in our SmoothWall.

Edited December 7, 2020 by smarties11

[mdrabble]

Currently running WinMTR to check for latency just incase there is an issue with ISP.

 

They said they have checked their end but worth running a check from our end.

 

Ok now for the numpty question time....... what is the difference between www. Google.co.Uk and google.co.uk in terms of running a traceroute?

Edited December 7, 2020 by mdrabble

[nathan3388]

Could be a faulty switch, if you are dropping pings to a certain area of the network could be a loop back on the network?. Have you spanning tree turn on might be worth it. To check the network.

[tom_newton]

Smoothie only needs conditional forwarders to your dc for local Lookup, you can and should probably point at all the 8s or whatever for general resolving

[mdrabble]

Bit of an update - might actually be an ISP routing issue.

 

We have had an outage since yesterday afternoon and after email tennis and phone calls from me, I hit the roof when I was asked to provide traceroute diagnostics to googledns - since then phoned 3 times and had the same conversation - mainline down but out backup connection is showing as up! - queue long repeated conversation leading them to suddenly realise there is a routing issue and we have no connectivity!

 

Still waiting on an update - will give them another hour before I call them again!

 

1st time this has happened in the 9 years with them.

 

On the plus side - been very quiet in the office this morning