Switches crashing almost daily!

[TomHD]

Hello All,

 

I'll say it again, hardware is not my strong point!

 

I have been suffering some issues/crashing with a couple of switches since half term. My solution till now has been to reset using the pin button on the front, this solves all issues for an undetermined amount of time! Alas, I'm now thinking the issue is not just a temporary glitch

 

The switches are both Netgear GS748T, not sure of age or how long they have been in action. One is the core with ISP, DC, VM host and edge switches attached (2 over SFP), it also has some clients too. The other is an edge with just clients attached. I have logged into the switches and I have noted that the core switch gets its IP from DHCP and is not statically assigned, whether this is a problem I'm not sure?

 

I noticed that when they have crashed I can remove cables yet the activity lights remain lit and flash showing activity?!

 

What are my options for troubleshooting this?

 

Many Thanks, TomHD

[sted]

have they got an air gap around them maybe they are just overheating?

[FN-GM]

Sound like a loop to me. You will have 2 network ports connected together somewhere.

 

Turn on Spanning tree on all your switches and it will disable the problem port. Alternatively starting looking for the loop, It will be most probably be at the edge where 2 data points are connected to each other. It can be a big job!

[JordanT91]

Can you log into the switch's management site? Maybe you could have a look at the logs?

I'm not an expert at network hardware but could it be the switch is being overloaded? Maybe there's a loop? Can you turn STP on at all?

[SHimmer45]

if you are looking for a loop you will need to isolate (break the link) between blocks / switches. it can be a pain to do on your own.

imp if the problem occurs and is random i would lean towards a switch being faulty.

when they are "faulty" can you ping / contact them?

[TomHD]

@sted, yeah plenty of airflow, the room they reside in is air conditioned at 19 degrees

@FN-GM, I did wonder if a loop had been made during the half term, we had plenty of contractors in and lots of IT has been moved/fiddled with! I have RSTP enabled on both the switches that are crashing. If it is not enabled on other switches can it still cause this problem?

@SHimmer45, No when they have crashed they are unreachable

 

Thanks

Edited November 11, 2015 by TomHD

[GuyJD]

I have similar Netgear switches here (GS724TP) you should be able to log in to it's web interface and enable 'Storm Control' that will monitor ports for links that have unusually high traffic flow and disable the port if need be, don't enable it for any up-link ports though as you may end up with switches not talking to each other.

[FN-GM]

@sted, yeah plenty of airflow, the room they reside in is air conditioned at 19 degrees

@FN-GM, I did wonder if a loop had been made during the half term, we had plenty of contractors in and lots of IT has been moved/fiddled with! I have RSTP enabled on both the switches that are crashing. If it is not enabled on other switches can it still cause this problem?

@SHimmer45, No when they have crashed they are unreachable

 

Thanks

 

You should enable RSTP on all switches for it to work properly.

 

Not sure if netgear does this, but on your ports (apart from uplinks) enable BPDU guard. BPDU are generated by Spanning tree, if one is received the port is shutdown install. It helps greatly with issues like this. Make sure its BPDU Guard and not Filter, they are the complete opposites!

[truebluesteve]

We had a load of those switches and on the whole they were very reliable but as they aged they seemed to be less so. Couple of things worked for us when we had similar issues to yours

 

1. Replaced the Gbic's (You don't say whether they use fibre uplinks or not)

 

2. Do a complete factory reset as opposed to a 'soft' reset

 

Even though it shouldn't be the cause of your problem, I would give the core switch a static IP at some point.

 

We have replaced all of our netgear switches (well, bar four of them) with HP ones instead.

[TomHD]

You don't say whether they use fibre uplinks or not

Sorry, the core has 2 fibre uplinks to 2 other GS748T in separate buildings.

 

Just had to reset again on the look for loops this afternoon

Edited November 11, 2015 by TomHD

[mikkydoos]

Sound like a loop to me. You will have 2 network ports connected together somewhere.

 

Turn on Spanning tree on all your switches and it will disable the problem port. Alternatively starting looking for the loop, It will be most probably be at the edge where 2 data points are connected to each other. It can be a big job!

 

 

@FN-GM has got it ! Some b*gger's chained two sockets together.

[JordanT91]

Sorry, the core has 2 fibre uplinks to 2 other GS748T in separate buildings.

 

Just had to reset again on the look for loops this afternoon

 

Have you tried updating the firmware? Find the version of your switch and get the files from here: NETGEAR Support | GS748Tv1 | Smart Switches

[truebluesteve]

Sorry, the core has 2 fibre uplinks to 2 other GS748T in separate buildings.

 

Just had to reset again on the look for loops this afternoon

 

I would have expected the problem to be there all the time if it were a permanent loop - although of course it could be an intermittent one if someone is maliciously doing it. Also in my experience the switch recovers when the loop has been removed after a couple of minutes. You could set a continuous ping to the switch and remove the cables from it one at a time and see if it comes back up.

 

If you want to try the Gbic option and need spares then let me know as I have a load of Netgear ones here (they don't work in HP switches)

[FN-GM]

I would have expected the problem to be there all the time if it were a permanent loop

 

A broadcast storm builds up after a length of time and gets worse and worse. Its not something that just happens instantly, that's why it won't be there all the time and why a reset temporarily fixes it.

[truebluesteve]

A broadcast storm builds up after a length of time and gets worse and worse. Its not something that just happens instantly, that's why it won't be there all the time and why a reset temporarily fixes it.

 

All I can say that in my experience that when someone loops out 2 network ports we see the effects pretty quickly, and they return just as quickly if the switch is reset. I'm not saying it isn't a loop its just seems odd that it appears to be happening so randomly

[m25man]

Broadcasts tend to begin when a host MAC/IP is unable to locate it's required destination MAC/IP this is generally a misconfigured setting somewhere.

A loop (if such a thing really exists) is caused when a packet of data crosses the same interface twice.

In my experience the most common cause of a "Loop" apart from malicious sculduggery are badly configured Trunks or LAGs where what should be aggregated links become separated.

 

A more common occurrence is incorrectly configured multi casting or one

I've posted several times before about regarding a known bug in Intel adapters that will go into multicast when the PC goes into standby.

This is more likely to be the source of your issue.

 

http://arstechnica.com/civis/viewtopic.php?t=1253571

 

https://supportforums.cisco.com/discussion/12296926/intel-i217-lm-nic-causes-broadcast-storm-and-high-cpu

 

http://packetpushers.net/good-nics-bad-things-blast-ipv6-multicast-listener-discovery-queries/

[Jamo]

All I can say that in my experience that when someone loops out 2 network ports we see the effects pretty quickly, and they return just as quickly if the switch is reset. I'm not saying it isn't a loop its just seems odd that it appears to be happening so randomly

 

Yeah I'd say it would appear relatively quickly I would have thought. General broadcast happens relatively frequently (ARP, NETBIOS and common) on networks and if there was a loop it would soon become incredibly noisy. Any logs on the switches?

[difinity]

If it's random, it's possible a power supply issue, I've recently repaired a load of similar Netgear switches that had capacitor issues.

[mikkydoos]

If it's random, it's possible a power supply issue, I've recently repaired a load of similar Netgear switches that had capacitor issues. [ATTACH=CONFIG]33213[/ATTACH]

 

Ooooh. Thats properly blown!

[FN-GM]

Yeah I'd say it would appear relatively quickly I would have thought. General broadcast happens relatively frequently (ARP, NETBIOS and common) on networks and if there was a loop it would soon become incredibly noisy. Any logs on the switches?

 

Depending on the switch, they might have the resources to handle a storm for quite a period of time. But as more and more storms happen they can then buckle.

[TomHD]

Hi All,

 

I am still suffering this problem. I have found all the switches on the network and have ensured that STP/RSTP is enabled on all switches. Do I have any more options for troubleshooting? Would you suggest I do a physical check for a loop?

 

My LM has suggested I call someone in to have an audit of our physical network, anyone have any suggestions? (we are Suffolk based)

 

Thanks, TomHD

Edited November 17, 2015 by TomHD

[FN-GM]

Hi All,

 

I am still suffering this problem. I have found all the switches on the network and have ensured that STP/RSTP is enabled on all switches. Do I have any more options for troubleshooting? Would you suggest I do a physical check for a loop?

 

My LM has suggested I call someone in to have an audit of our physical network, anyone have any suggestions? (we are Suffolk based)

 

Thanks, TomHD

 

Have you enabled the BDPU guard on the access ports?

[mikkydoos]

Hi All,

 

I am still suffering this problem. I have found all the switches on the network and have ensured that STP/RSTP is enabled on all switches. Do I have any more options for troubleshooting? Would you suggest I do a physical check for a loop?

 

My LM has suggested I call someone in to have an audit of our physical network, anyone have any suggestions? (we are Suffolk based)

 

Thanks, TomHD

 

 

A physical check is definitely in order. Someone has plugged 2 sockets together. I bet my @$$ on it.

 

If you dont find a looped socket, check any cabs that aren't locked.

[difinity]

Hi All,

 

I am still suffering this problem. I have found all the switches on the network and have ensured that STP/RSTP is enabled on all switches. Do I have any more options for troubleshooting? Would you suggest I do a physical check for a loop?

 

My LM has suggested I call someone in to have an audit of our physical network, anyone have any suggestions? (we are Suffolk based)

 

Thanks, TomHD

 

And also a physical check inside the switch. The netgear are easy to get the top off.

Edited November 17, 2015 by difinity

[TomHD]

Thanks guys, will make the rounds this afternoon after sorting out a projector and a laptop

 

Regarding BDPU, there are no options for this in the web interface.

 

TomHD.