Network wide connection dropping

[Qegs]

Greetings all,

 

Over the past 2 weeks we have been having some serious issues with our network, where what appears to be random connection drop outs.

 

We are running a Windows Server 2008 R1 network with some Windows Server 2003 servers as well. All of our client machines are Windows XP Pro with the exception of 4 admin machines which are running Windows 7 Pro.

 

We have 3 Domain Controllers (Windows Server 2008 Core), we also have 2 Servers running Hyper-V with several virtual servers running on each of them. All of our servers have network Teaming enabled as well.

 

We also have a Ruckus Wireless network.

 

Our Main switch is a Procurve 4203VL and we have approx. 10x ProCurve Switch 2610-24 dotted around the school.

 

The Drop outs at first would happen network wide, but now it seems to be randomly happening in different areas.

 

We are running out of ideas on what could be causing this and hope that someone here may have a solution or idea?

 

Thanks

[smithson83]

Does it follow a class around, maybe its a little **** cross patching, little darlings Edited September 23, 2011 by smithson83

[Geoff]

Do you have STP, BDPU and loop protection setup on your procurves? If not you should. Here's a PDF.

 

http://h40060.www4.hp.com/procurve/uk/en/pdfs/application-notes/How_to_improve_and_harden_spanning-tree_configuration_Configuration_note_Dec_08_A4.pdf

[DAZZD88]

@Geoff handy link that. Doubt we'll ever need to use it but it's good to have just in case we get the opportunity.

[pantscat]

Nice link from @Geoff there.

 

Your Procurve kit should give you a pretty good idea of what's going on - check out the logs on the switches.

[pete]

Have you got anything monitoring the links so you can record when and where it happens? i.e Nagios, Cacti, ping script that logs somewhere?

 

What do your per-port counters say on the backbone switch? Do any particular ports have high error counts? Same for the edge switches? See if you can track down a recurring theme.

 

Are the switches on UPS? Surge-protected? Just basic mains power?

 

Is anywhere immune from the drops?

 

Also YGPM.

[zag]

I've had exactly the same problems as you describe before.

 

I enabled STP on all our netgear switches at the time, not sure if it helped though.

 

It turned out to be the wireless controller overloading the network. We found out the students were trying to break the key with hacking software.

 

At the time it was quite dificult to troubleshoot as the wireless was needed all day. But start changing things one by one and you will soon find it

[zag]

Oh forgot to say check out my php server monitor in my signature. If you set it up and set it to auto refresh every 30 seconds you will see ping times increase usually before a network outage.

 

Then you can track the problem on the switches usually before they go down.

 

Doing a ping -t will work as well

[Qegs]

Thanks for all the input, it has given us a few more directions to move in, I will keep you posted.

 

Thanks

[Qegs]

Hello All,

 

Just an update on our situation, everything seems to being running fine now, we think we found what was causing the problem. One of our switches was flagging up alot of errors on a particular port (32) when we went to look at this port we couldn't work out want was connected to it causing all this traffic and errors. So we ended up pulling the cable up out of the trunking and it turned out the the cable wan't actually connected to anything.

 

The cable looked like it have been cut and left in the trunking, and from what I can tell all of the cores of the cable were touching each other. The funny thing is that cable was left like that before I started working at the school which is coming up to 4.5 years. So why it has only recently started causing a problem I don't know.

 

But thanks from everyone that got in contact.

[Geoff]

Thou shalt audit thy network cabling!