HP Managed Switches: Detecting and dealing with a loop/broadcast storm

[talksr]

Hello there,

 

I have a set of 9 managed HP switches running in a stack.

I have reason to believe we have a loop on the network. I enabled spanning tree protocol on all switches in our stack and almost immediately, the network went back to normal behaviour.

 

What is the best way for me to identify which port is causing this problem?

I believe HP switches have a fault finder feature which may help, but I don't really want it blocking ports as this could cause even more problems on the network, particularly if it blocks suspected ones which are not actually causing a problem.

 

Any suggestions or ideas would be greatly appreciated.

[Tefters]

Wait till after school and turn off STP so your network starts hanging, pull 1 switch at a time from the stack until you find the problem switch and then isolate port by port, best thing about loops is a simple ping -t on a laptop next to you shows the issue in play

[talksr]

Wait till after school and turn off STP so your network starts hanging, pull 1 switch at a time from the stack until you find the problem switch and then isolate port by port, best thing about loops is a simple ping -t on a laptop next to you shows the issue in play

 

I had thought about this, but we have over 250 cables. It will take hours!!

[Tefters]

Talking an hour tops surely? This is how I would do it (and have the last 3 times on BIG setups):

 

1. Turn off STP and have laptop next to me pinging something like a DC ensuring the issue is there aka pings all over the shop and not request timeouts.

2. Start pulling switch by switch in the stack til the issue goes away and everything is stable, takes like 20/30 secs to stabilise when I last did it.

3. Start pulling cables in the problem switch identified above in groups of 10's until you find the group then start working down from there.

 

It's tedious but actually really simple and relatively quick once in the swing of it

[IrritableTech]

I had thought about this, but we have over 250 cables. It will take hours!!

 

It shouldn't really and I presume you'd like to fix the issue as you came here.

 

Disconnect a switch for a minute and look at the activity of the others - has it decreased? No - then reconnect it and move onto the next. Yes - then start working through individual connections on that switch.

 

You could look in the GUI of each switch to see if you can see any which have usually high traffic on them.

[Tefters]

I love these sorts of issue from time to time, beats sitting in my chair bashing my head against the desk over Microsoft and application logs or your usual hunting down reg settings for stupid blocks/issues.

[alfatec]

I presume your switch's are HP Procurve if they are managed. If they are you can enable Loop Protect which is a seperate thing from Spanning Tree. Take a look at this link https://wiki.xdroop.com/space/HP/ProCurve+5412z/Loop+Protection

if you enable this it will show you exactly which port is looped.

[talksr]

It shouldn't really and I presume you'd like to fix the issue as you came here.

 

Disconnect a switch for a minute and look at the activity of the others - has it decreased? No - then reconnect it and move onto the next. Yes - then start working through individual connections on that switch.

 

You could look in the GUI of each switch to see if you can see any which have usually high traffic on them.

 

Yes will try, but we have so many in the rack, its is jam packed, its hard to see the lights flashing for the wires in the way.

 

I love these sorts of issue from time to time, beats sitting in my chair bashing my head against the desk over Microsoft and application logs or your usual hunting down reg settings for stupid blocks/issues.

 

I agree, but I am on site very little, time there is very precious, although without this issue resolved, it will be tricky to get anything else done.

 

I presume your switch's are HP Procurve if they are managed. If they are you can enable Loop Protect which is a seperate thing from Spanning Tree. Take a look at this link https://wiki.xdroop.com/space/HP/ProCurve+5412z/Loop+Protection

if you enable this it will show you exactly which port is looped.

 

Yes, all Procurve. Thanks for your helpful suggestion, will take a look now.

[talksr]

Very odd.

I used the loop protection that we have on our Procurve hardware and it is detecting nothing across our switches.

I tested it on one which is not on the network and found that it was unable to detect a loop with STP enabled. Once I ran no spanning-tree and re ran show loop-protect all, it had immediately identified the two ports and had disabled them.

 

When I went through our live switches on the network, nothing is being detected. I have started checking cables and trying by unplugging some on our switch, but still, no change. The lights are all flashing as expected, not the usual loop related flashing sequences.

If I ping our server, this is the response I get, so something is still wrong here: