Spanning tree topology changing on half of our switches multiple times per hour?

[Garacesh]

Something very bizarre is happening with spanning tree for about half of our switches with Spanning Tree and I'm struggling to pin it down.

 

Periodically, typically 2 or 3 times per hour, spanning tree topology changes on about half of our switches.

The other half of our switches have more sensible timespans, 22 days, 56 days, 197 day,

 

I've been through looking at all of our switches. As it currently stands:

• All of the switches have the core switch as their CST/Root source.
  
• All of the switches have themselves set as the IST/Bridge.
  
• Every switch that can is running MSTP. One is running RSTP because it's a cheaper switch that doesn't have that option.
  
• All of the Cisco-style (tagged/untagged) switches have CST priority 16384
  
• All of the Cisco-style switches have IST priority 32768
  
• All of the Procurve-style (access/trunk) switches have CST priority 16384 (except the one that can't run MSTP)
  
• Most of the Procurve-style switches have IST priority 24576
  
  • The only exceptions here are the old cores that have IST priority 32768
    
  • And the current core (the one that's the CST/Root for all the others), which has IST priority 16384
    

  [*]Topology changes occur on all of the the affected switches simultaneously. Last topology change occurred changes more-or-less in tandem.

 

Additionally, if I log on to any of the affected switches and look at the logs for when the topology changes, there is... nothing. Absolutely nothing. No events correlating with the time the topology changed.

Weirdly, not ever switch in a chain does it. We have one boarding house where the 1st switch over there will do it, but neither of the 2 switches down the link from that one do it.

 

I'm massively at a loss now. Where the heck do I go from here? I've not got a clue.

[Davit2005]

Dodgy link between switches?

 

We saw the same between 4 of our cores that use MSTP for redundancy in a mesh formation. The preferred spanning tree route based on priority was changing regularly within an hour.

Edited September 1, 2023 by Davit2005

[FN-GM]

Have you checked that spanning tree is only occurring on links between other switches?

 

Maybe there a rouge switch (or something else) connected to one your access ports that is trying to participate in your spanning tree topology. To prevent issues like this I always enable the BPDU Guard on all ports apart from uplinks.

Edited September 1, 2023 by FN-GM

[Garacesh]

Maybe there a rouge switch (or something else) connected to one your access ports that is trying to participate in your spanning tree topology. To prevent issues like this I always enable the BPDU Guard on all ports apart from uplinks.

See I was thinking maybe something like that but there's just no pattern to which switches do or don't change themselves.

My initial thought was that a rogue device could be the issue and I'd be able to find roughly where the device was by looking at the cost/weight, but it truly is a random assortment of switches (even including the current core itself)

 

We did discuss this BPDU with our 3rd party support a while ago. For the procurve-style switches we've currently got bpdu-protection and admin-edge-port set on ports connected to access points, but that's it. Their recommendation was to enable it on access points, the line out to the internet connection, and any point-to-point links (except all of our point-to-point links are between switches, so turning that on kills the link)

Haven't gone through the cisco-style ones yet, but looking at their configs, they all just have spanning-tree with nothing else, except one switch that has an additional spanning-tree priority 6.

[dmj]

Surely you want to have all the switches have the same priority, except the core with a lower one?

[Garacesh]

Surely you want to have all the switches have the same priority, except the core with a lower one?

 

I would have expected so, at least that was my assumption, this is all a bit of a learning experience! I'm going through our infrastructure tidying up and documenting for handover.

I've done plenty of vlans, rate limiting, snmp, dns, etc in the past but this is a bit of a deeper dive than I'm used to.

[dmj]

If yours is a simple star without failover links I would set edge switches at default and core lower (at 24576). If you have failovers (eg via another physical building) you may need to set the near edge at 28672.

(it was about nearly 10yrs ago since I last did failovers on networks!, so don't quote me on the last bit)

[Garacesh]

Slight correction, the cisco style switches all have spanning-tree priority 6, for some reason Office has lost half of my documentation It's lost the dead end VLAN I set up too, no idea why. Uuurrrgh gonna have to update them all over again. Grumble.

 

The network is basically a star, but we're currently in the process of stacking 2 separate switches into one logical switch over the network. We have a 25GB fibre link between them as they're in different buildings, but I'm fairly certain they're still operating independently.

 

Amusingly, the main core is affected by this issue, but the other half is the one with 197 days on it.

Edited September 1, 2023 by Garacesh

[dmj]

Slight correction, the cisco style switches all have spanning-tree priority 6, for some reason Office has lost half of my documentation It's lost the dead end VLAN I set up too, no idea why. Uuurrrgh gonna have to update them all over again. Grumble.

 

Infrastructure as Code FTW