Intermittent Monitoring Issue

[ITGURU]

I have monitoring setup on network to ping all switches and core devices and alert on unreachable/ reachable.

 

I've purchased a qty of new switches and monitoring keeps reporting randomly the device being unreachable.

 

I've taken it up with support and they say it's a network issue but the same issue with the same model switches happens on all our networks where I've installed this particular model switch.

 

the weird thing is the IPs monitored that are connected to the device maintain 100% uptime but the poe switch IP has the sporadic outages and ping drop.

 

anyone any ideas on this as I think it's a hardware revision issue as have older revisions on the same model switch on the network and don't experience this issue and no other devices monitored present this issue.

 

The only discovery I've made is that when watching the ping from cmd after the 1 2 or 3 request timed out the next response time is then over 1000ms.

 

However at the point ping times out to the switch I still get full ping reply from devices connected to the switch.

[FN-GM]

As it all happened to be the same model, maybe a software bug, is the firmware up to date on these switches? Edited January 15, 2023 by FN-GM

[dmj]

As you've discovered, ICMP isn't the best method for monitoring: You are trying to measure availability of the service and using ICMP as a proxy to measure whether a port responds to an unrelated protocol. i.e. it's not a direct measurement of what you actually care about and uptime is not the same as availability.

 

There could be a bunch of reasons why the ICMP fails intermittently, QoS? duplicate IP? routing issues? Firmware? STP convergence? or just that the network is loaded and dropping some ICMP. My advice would be to implement some SNMP checking for something that you actually need to monitor ( uptime, throughput etc) and not whether the your monitoring receives an ICMP response. The SNMP metrics might also give an insight as to why the ICMP is failing.

[ITGURU]

As you've discovered, ICMP isn't the best method for monitoring: You are trying to measure availability of the service and using ICMP as a proxy to measure whether a port responds to an unrelated protocol. i.e. it's not a direct measurement of what you actually care about and uptime is not the same as availability.

 

There could be a bunch of reasons why the ICMP fails intermittently, QoS? duplicate IP? routing issues? Firmware? STP convergence? or just that the network is loaded and dropping some ICMP. My advice would be to implement some SNMP checking for something that you actually need to monitor ( uptime, throughput etc) and not whether the your monitoring receives an ICMP response. The SNMP metrics might also give an insight as to why the ICMP is failing.

 

It isn't a discovery - i monitor 4 physical sites all in exactly the same way and no other devices show this issue. Firmware is all up to date and engineers have remoted in to capture with wireshark to see if they can see whats happening. these are literally 8 port poe switches and happens even if no devices are connected to the poe ports bar an uplink cable.

[dmj]

It isn't a discovery - i monitor 4 physical sites all in exactly the same way and no other devices show this issue. Firmware is all up to date and engineers have remoted in to capture with wireshark to see if they can see whats happening. these are literally 8 port poe switches and happens even if no devices are connected to the poe ports bar an uplink cable.

 

well, look into the issues I suggested then. Perhaps if you were actually monitoring the network you'd know the reason already.

[ITGURU]

well, look into the issues I suggested then. Perhaps if you were actually monitoring the network you'd know the reason already.

Its definitely a firmware fault so have gone back to vendor. the switches ping but now monitoring shows loss of http access to the gui on one of the switches so not using ping would not be a real account of whether online or not as snmp failing now on one switch but still pings ans connected devices still accessible.

[jthompson]

I've seen this with some old Netgear switches in the past. PING and SNMP response fails, so it appears as dead on the monitoring platform, although the device continues to perform as switch for connected devices okay. My guess was that the management part of the device would crash periodically, but the rest of it continued to work okay.

[ITGURU]

I've seen this with some old Netgear switches in the past. PING and SNMP response fails, so it appears as dead on the monitoring platform, although the device continues to perform as switch for connected devices okay. My guess was that the management part of the device would crash periodically, but the rest of it continued to work okay.

 

Yes that is my result from the weekend as has now lost access to the GUI but devices and ping still working therefore is a firmware issue.