Ping strangeness

[truebluesteve]

Ok so this is an odd problem which has left me scratching my head...

 

We use a program called iPMonitor which periodically pings all our switches (and servers) for availability and sends an email if if cant reach device after a number of tries. Just recently we have had a number of occasions where it has indicated that the majority of our switches are, but subsequent investigation has found that they are fine. It happened again this morning and its an INSET day here so the network is very quiet so I have carried out a couple of tests

 

- I can ping all the switches that are reported down, from my PC as can my colleague from his

 

- If I try to do same from our servers (both physical and virtual) I can't reach them

 

- The switches are in different physical locations but are all connected to the same core switch

 

Over the course of the day the switches will come back 'online' as it were, and all will be OK until the next time it happens

 

All the switches are HP - V1910's and Procurve 2510's and it would seem the issue seems to affect the former but not all of them. Perhaps a firmware update is needed although that still doesn't explain why I can ping them from my PC!

 

 

I'm a bit baffled so any suggestions would be great!

 

Cheers

 

Steve

[cullingsh]

We use the V1910s and there is a recent firmware available, released end of October this year - I've put it on a few switches as a test to ensure it runs ok before rolling out site-wide.

 

Do all of your switches have static IP addresses? And do you have VLAN's set-up and running ok?

[truebluesteve]

We use the V1910s and there is a recent firmware available, released end of October this year - I've put it on a few switches as a test to ensure it runs ok before rolling out site-wide.

 

Do all of your switches have static IP addresses? And do you have VLAN's set-up and running ok?

 

Yep they all have static IP's and the VLANs all seem to be fine.

 

Was the firmware update OK?

[cullingsh]

Seems to be - it was only late last Friday it got applied, but had no calls yet reporting problems.

[peterp]

We've seen things like this happen when there is a network loop. Packets go missing, pings fail etc.

Another possibility is 2 devices with the same address (is it disallowed in DHCP)

[ZeroHour]

It could be a screaming NIC which is only getting turned on for specific classes etc. I would write down the times and see if you see any patterns emerging like the same rough time each week etc in case the faulty kit is only turned on for a limited time.

[truebluesteve]

Yeah.. I will start doing this although so far there doesn't seem to be a pattern that I can see. Its happened 3 times on different days and times which isnt that helpful!

[ZeroHour]

How long roughly would they be unpingable for?

[truebluesteve]

How long roughly would they be unpingable for?

 

Well it varies - it can be a few minutes to a few hours and not all of them will come up again at the same time. During this time people can still log on and do whatever they need to on the network

[jonemery99]

Are the servers running Broadcom nics on smart load balancing? Used to have weird issues with them and hp switches dropping pings if they were not trunked

[truebluesteve]

Are the servers running Broadcom nics on smart load balancing? Used to have weird issues with them and hp switches dropping pings if they were not trunked

 

Not really sure, but I have a feeling all our hosts are using Intel NICs

[sonofsanta]

We used to have problems with certain lower end 3COM switches wherein the ARP table on the switch would get full (they were only 256 big) and so not every machine would get a ping reply - e.g. I would get no reply & panic, then get a tech to check and it'd be fine. Going in and clearing the arp table from the command line would fix it. I've a feeling the V1910 is a rebadged 3COM so I'd hazard a guess this is your issue. When it happens next, putty/telnet in and

reset arp dynamic

If you can ping from everywhere again then you know that was the problem, so back in your putty session:

sys
arp timer aging 5

to reduce the time the switch holds on to its ARP entries, so it doesn't fill up.

 

Other than causing panics, I don't think it caused any real issues, and it's stopped happening since we VLANned into segments < 256 hosts big.