Network being throttled?

[smurfomatic]

Oh, the joys of the new school year...

 

Staff came back in for inset on Monday, all seemed fine until just after lunch and then the network ground to a halt - and still isn't right.

 

Wireless devices struggle to connect as they're not being issued IP addresses quick enough and the DNS resolution tests from the Smoothwall are reporting delays - around 4 seconds. However, nothing's changed on either DHCP or DNS so I don't think that's the issue.

 

It's as if there's a device "spamming" the network with broadcast messages (had this before with a faulty WAP) but running Wireshark doesn't show up any unusual activity.

 

I've asked all staff to shut down their devices as they left today, and while there's still a few connected (we're a boarding school and have some resident staff) over 90% of devices are off the system and it's still running like a dog!

 

Even pinging between two servers is getting a failure rate around 60%. Also, the Ruckus zone director keeps losing contact with its access points because it can't get the data through.

 

One major concern of mine is that we have a company in installing new central heating - I'm hoping they haven't damaged a cable while drilling. That wouldn't account for issues between devices a lot closer together though - for example, the server in the teaching area struggles to communicate with the Smoothwall in the same room.

 

I have tried:

 

Restarting all servers

Restarting network switches

Rebooting Smoothwall and Zone Director

Disabling DNS on servers alternately

Clearing DHCP leases

 

Please tell me I've not missed something obvious...

[plexer]

Network loop?

 

Ben

[Liam]

Have you got a loop somewhere on your network? Have you enabled loop protection on switches? Stp?

 

Sounds to me like you have a loop to be honest.

[Mark182]

is spanning tree enabled if you have Cisco switches?

 

Ah, did not see above post!

[smurfomatic]

Thanks Ben & Liam - I did think it may be a network loop, I need to go round and check all the cabinets tomorrow (you know, the ones no-one should be fiddling with) to make sure the cabling hasn't been messed with.

 

I can't remember if loop protection is turned on, we had a major network upgrade this time last year which I was supposed to supervise but I had a ruptured appendix instead...

[smurfomatic]

Mark - thanks, but we've got HP Proliant gear rather than Cisco.

[plexer]

You can still have spanning tree for the hp kit.

 

I would think it's a loop of a couple of desktop ports rather than at the cab level.

 

Ben

[smurfomatic]

I'm expecting it to be the same Ben, just don't fancy having to check all those lovely new desktop ports that were fitted last September...

 

[gets out network map and biro]

[rad]

we had a duff network AP connected into a switch, might be worth checking what is plugged in classrooms

[smurfomatic]

thanks rad - looks like I'll be doing a full tour during tomorrow anyway, may as well check for anything else plugged in while I'm there!

[Liam]

Easiest way to do it is to isolate switches if you don't rely on them as part of your back bone of course . Then you can narrow it down. Defo worth enabling loop protection, and it's really straight forward on the cli with hp kit.

Have you no vlans?

[FN-GM]

I would check the syslog on your switches. Im not sure about HP but Cisco you can't see them on the console via SSH or Telnet by default. You may have to plug a laptop in directly. It may give you the exact info you require.

 

Do you have VLANs? If so does it impact all of them? From what you have said im guessing not.

 

Thanks

[smurfomatic]

We have no VLANs set up, I will have to look into the logs directly I think - hopefully this will narrow down the search...

[FN-GM]

You can turn it on so you can see the logs on the console via SSH or Telnet. However for HP i don't know the commands.

 

For Cisco you use the "logging monitor" command on the VTY lines. Not sure how close HP might be to that.

[IanT]

Log onto each switch via CLI and do "show loop-protect", to look at the logs on the switch do "show logging"

 

If spanning tree or loop-protect are not on, get it configured.

[smurfomatic]

Sorted! Found the port causing the network loop and disconnected it - not sure who it belongs to yet, but they'll start complaining once they realise everyone else is back online...

 

Panicked when I couldn't get hold of a USB to serial adapter to plug in the console cable, fortunately the state school just down the road had one they lent to me.

 

Thanks for all the advice, really helpful.

[IanT]

Might be wise to configure the loop protect on your switches

[FN-GM]

I would configure Rapid Spanning tree on your system. After that put the BDPU guard on your ports that connect to the end users. BDPU's are sent out by spanning tree to detect loops. If one of those ports with the guard picks up the BDPU (from either one of your switches or another thats plugged in) it will disable the port instantly.

[smurfomatic]

Ian - loop protection switched on as I went round!

 

FN-GM - I will be setting up spanning tree, BPDU etc. once I get a chance to cath up with everything else I should have been doing. Oh, and I'm ordering a USB to serial converter as well...

[IanT]

Good guide

 

BPDU Protection on HP ProCurve Switches — Evil Routers

[rad]

@smurfomatic so what was it?

[smurfomatic]

IanT - thanks, that looks really useful. I might even get a chance to read it some point soon!

 

Rad - I'm assuming it was a network loop. Loop protection on one of the switches picked it up on one port, unplugged that cable and all is well. Haven't been able to trace back the port to find out where it is, but no-one's complaining about loss of service yet...