High latency on wired LAN

[Frodo_Baggins]

Intermittent latency with high ping, pings on my machine are all over the place ranging from 5ms to 300ms (Reported by end users as slowness when accessing websites)

 

Speedtest on the machines isn’t great and shows download latency go over 3000ms (overall speed is good but high ping latency)

 

Equipment:

Router is Mikrotik provided by ISP (replaced yesterday as believed to be faulty)Core is 2x Aruba 5406Rzl2 Edge switches are Aruba 2930F

 

Router>Core>Edge Switch >End User Device (Server 2019 DC does DHCP & DNS, DNS uses forwarders from ISP)

 

Routing: Core uses router as gateway, core has VLAN’s setup with a static IP that acts as the gateway for vlans on edge switches e.g. AP gets plugged into port 4, port 4 on the edge switch is set to VLAN 4, VLAN 4 has a gateway of 10.1.4.1 which is the IP of the VLAN on the core, gateway for this is the router.

 

Port setup on Mikrotik to core which acts as a transit network to internal range

 

- - - Updated - - -

 

Strange things that are related:

 

When pinging another device on the network that connects to another switch it shows TTL of 118

 

Port connected from core to router as a transit network has tx errors which looks to be increasing.

 

Next step is to replace the cable going from router to core and setup snmp monitoring, anyone had similar issues and have an idea on where to start looking?

Edited February 14, 2023 by Frodo_Baggins

[themightymrp]

My first query from your screenshots is "Wifiblaster". Who is this? The only ISP I can find is one in Spain, so do you have some kind of VPN set up?

 

Pinging another device on your domain, unless you have some custom settings configured, would normally come back as a TTL of 127. However, whatever is providing DNS lookup can contain a DNS record to adjust the default TTL. Try pinging Google on 8.8.8.8 and you'll get a TTL of 118

 

I'm thinking something DNS is possibly your issue. Are you able to change the DNS server IP on something and try speedtest again to see if it makes a difference?

 

Alternatively, it could be a routing issue. From one device on the network, open CMD and run tracert to the name of device 2. Eg, tracert computer2 - you would expect after a short delay you would just get one router in the path (your onsite one). If there are more in the path, your traffic is going out to come back in indicating something wrong in the routing table

[dmj]

Pinging another device on your domain, unless you have some custom settings configured, would normally come back as a TTL of 127. However, whatever is providing DNS lookup can contain a DNS record to adjust the default TTL. Try pinging Google on 8.8.8.8 and you'll get a TTL of 118

 

Your're confusing ICMP TTL with a DNS Record TTL

In ICMP each hop reduces the packets TTL until it reaches zero and gets discarded. It's got nothing to do with DNS TTL records which is how long the record lasts in cache before it requests a new one.

 

This issue doesn't sound like a DNS problem to me.

[DavR]

Sounds like a loop, to me. Unpatch chunks of your network in turn, until the high latency stops. A healthy wired network shouldn't see more than a few ms at a time on ping responses.

 

Maybe strip it down to just your device, one switch, your servers and the Internet connection, and see what performance is like then, and build back up from there.

[Davit2005]

Check Internal pings then external pings but could be a loop. If you can check external pings from the router too or the core switch might help isolate the issue.

[rob_coles]

Are you using Sophos at all?

 

Sophos has a issue with Speedtests.

 

I’d be tempted to cli version of Speedtest to check results.

 

 

https://www.speedtest.net/apps/cli

[PotNoodleTech]

Have you thouroughly tested internal pings around the place - i.e is it only internet latency and bad pings you are talking about and the lan to lan is all fine?

[kevin_lane]

Yea I think it it was me I would most probably connect to core router and see what is happening, and then work backwards, (I think someone else said the same thing)

 

Also don’t forget you have the command pathping which will help

 

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/pathping

 

But also when did this start ? Did this happen with the old router ?

 

Have you made any config changes to any of the switches recently.

 

 

Sometimes cabling can be an issue however if you have decent cat5e and not any self made (not saying self made are not good) but if you have t crimped them correctly your going to get issues.

 

 

 

It thin once you have identified where the issue is you can look at checking the rest of the stack.

[Frodo_Baggins]

Thanks everyone, half term for me so time to start properly looking at this! ISP is Talstraight - they have said pings from the router externally look fine.

 

 

The aforementioned port going from the core to the router is showing a high collision rate so will swap the cable shortly in case it's dodgy

 

 

Checked loop protection and It's enabled on all but the core, no loops detected but don't know how trustworthy that is

 

 

After swapping the cable I'll be updating firmware and enabling loop protection on the core, If nothing is detected I'll start turning off switches across site and turn on till I find it.

 

 

Only recent change was an extra port being setup on the router and change to DHCP for the gateway for our AP's (reverted this change in DHCP but still occurring.)

 

 

From my PC

 

 

Ping to VM 1-8ms TTL 127

Ping to core 1-64ms TTL 255

Ping to another machine on same switch as i am connected to currently 1-13ms TTL 127

 

 

From Core

 

 

Ping to VM 1-2ms

Ping to another machine on switch round the corner 1-3ms

Edited February 20, 2023 by Frodo_Baggins

[chaplic]

Think it's just gonna be a divide-and-conquer, split it in two, test one side, test the other and so on... I've seen things like a dodgy network card just spraying electrical noise out causing crazyness..

[robintech]

had some odd problems where a gateway device was spoofing MACs for some reason, the arp table on some computers had the wrong entry for the gateway IP but were somehow finding a way out onto the internet at a high latency

[PotNoodleTech]

Thats not good - internal pings on a modern network internally even across multiple switches should be 0ms if your getting anything more than 2 then there's something not right IMHO.

[Frodo_Baggins]

Started turning off switches this morning, still 12 to turn off did notice something weird in DHCP which I deleted yesterday and has came back but with a short lease and TTL OF 63 so there's hope still! (Managed 3COM 8769 which AFAIK we don't use for anything)

Lease on the 3com device is like an hour each time it shows in DHCP

Edited February 21, 2023 by Frodo_Baggins

[PrimaryNetMan]

Thanks everyone, half term for me so time to start properly looking at this! ISP is Talstraigh

 

There's your problem.

[chaplic]

Started turning off switches this morning, still 12 to turn off did notice something weird in DHCP which I deleted yesterday and has came back but with a short lease and TTL OF 63 so there's hope still! (Managed 3COM 8769 which AFAIK we don't use for anything)

Lease on the 3com device is like an hour each time it shows in DHCP

 

We're all with you....good luck and happy sleuthing!

[sshabir]

Did you manage to figure it out then?