Possible storm/loop or failing switch.

[chrbb]

Twice this week our network has suddenly failed, server still running but all computers on the network unable to access shared drives or the internet. 'Fixed it' yesterday by randomly poking around patch cables in the switch cabinet, but today's failure took alot longer to sort. It also took out the Ruckus wireless network, the switch for the access points is in the same comms cabinet. Everything finally burst into life when I removed one patch cable from one of the 3 switches, it's light was flashing slightly differently from all the others and when I took the cable out all the switch lights when crazy but the whole network started working again.

 

Other than the access point switch, all the others are bog standard unmanaged switches, what's the best way to check for a failing switch or loop? I'm going to send tomorrow morning taking all cables out and replacing them one by one, but wondered if there was an easier way.

 

Had to smile through the blood, sweat and tears this afternoon, the entire school staff suddenly become experts on networking and the internet at the first sign of a problem and offer many possible solutions

[andydis]

I don't believe you can do anything unless you have managed switched , the unmanned ones have no intelligence over storm handling or spanning tree ,

 

I know that's not helpful or what you wanted to hear .

 

Perhaps ask the powers that be about upgrading them ?

They should listen to you as

It's quite serious if a student could bring down the whole school by simply plugging in a cable wrong

[win]

You really need to change to managed switches. Although procurves are ideal even a cheap unknown branded managed switch with STP enabled would be better, certainly safer. Second hand ones on ebay could save the network from a nasty loop!

[cullingsh]

I would go look at what is on the other end of the cable you unplugged - make sure someone hasn't put a patch cable between two network points or done something else to it as a short term fix.

 

Then look at upgrading the infrastructure as a more longer term aim

[pete]

Even if you only upgrade the core switch(es) to managed versions it'll stop problems on edge switches getting beyond that edge switch.

 

eBay (or someone like ict-direct) will get you functional second-hand Cisco* or HP Procurve managed switches if money is tight.

 

*people tend to sell off their lab kit when they complete a Cisco course.

[MStainton]

You could also try IT in stock and Easy IT for cheap 2nd user switches.

[m25man]

To be honest it's unlikely to be one of your cheap unmanaged switches that is to blame more likely that they are just unable to stop or isolate the source.

Throwing ££££s at the issue by installing managed switches just to use STP to try and fix the problem is not an answer.

 

STP only works by blocking a link when it sees the same packet pass through the link more than once.

It's used in conjunction with OSPF to manage intentional managed redundant loops.

 

Your issue is most likely a broadcast or multicast storm, your best way to solve the issue is to monitor and capture.

 

Even the more basic web managed switches have GUI with port stats that can show how many errors or dropped frames have occurred on a specific port. If you have anything like that on your switch that's where you should start.

 

For the last year or so there are well documented issues with bad drivers and Intel NICs that cause them to spew out multicast packets when they drop into standby, this is a more likely cause.

 

I can count the amount of switches I've had to condemn in the last 10 yrs on one hand, most blackouts tend to get traced back to a specific node(s) or bad configurations.

[mikeyd101]

try monitoring with something like wireshark that might lead you to some other information like ip or mac address of the offending packet spewer.