Intermittent Network Issue

[IAMCloud_Curt]

Morning all,

 

Got a weird one here for you all, the past few weeks, half the school is having really bad intermittent network faults.

 

By this I mean, when browsing the network drives, opening sims and logging on it's taking longer than usual.

 

I started to begin a little experiment, the Headteacher came and saw me and said her printer say's it was Offline. This was one of the rooms being effected by this problem, so I went down and pinged it expecting no reply as print server said it was offline.

 

However, I got a few replies, then time out for ages. I did the usual, changed ethernet, changed network port. It's a double network socket so I swapped her laptop with her printer, still the same.

 

Started to scratch my head a little, and went back to basics. I pinged our gateway from both network ports in her office. Timing out all the time, wen't back to my office and I don't receive any time outs.

 

I've attached screenshots of ping results, any ideas on what this could be? It's happening in about 9 rooms now on one side of our school, the other side is fine (runs off different cab)

 

Any help is really appreciated, staff are beginning to get frustrated.

 

[Steve21]

Sounds like a network loop on that part of the VLAN etc. (Although looks like a low ping for that if it was, unless it's higher up chain?)

 

Is your office on a different one to them/that area?

 

Steve

Edited November 17, 2015 by Steve21

[IAMCloud_Curt]

Yeah my office is upstairs, upstairs is fine as it's running on a different cab. Only reason I pinged from my computer was just to check everything was ok.

 

Best cause of action with network loop?

[Steve21]

Well if it is one, it's generally a bit of brute force. Depending how many switches etc you have in that side/floor.

 

Lets say you got 4 switches there. With one being the uplink.

 

Try connecting directly from uplink on laptop. Good pings?

Re-add one switch, and connect laptop to switch port. Good pings? etc etc

Until you can look at what switch is causing a problem.

 

Can you login to the switches config? As without unplugging anything might be able to see if it's a switch issue from dropping ports/connections etc

 

Steve

[ChrisH]

Don't suppose you have managed switches? The log on them would be the first thing to check otherwise depending on your physical topology you need to keep testing but start unplugging links and or whole switches until the timeouts dissapear then you can isolate it to a particular link or switch then you start unplugging the cables on that switch until you see an improvement.

[Michael]

So can you isolate the issue to a particular switch? And have you tried rebooting them out of hours if they have a high uptime?

 

If you haven't already, you should enable Spanning Tree, as this will isolate the loop regardless.

[truebluesteve]

Well if it is one, it's generally a bit of brute force. Depending how many switches etc you have in that side/floor.

 

Lets say you got 4 switches there. With one being the uplink.

 

Try connecting directly from uplink on laptop. Good pings?

Re-add one switch, and connect laptop to switch port. Good pings? etc etc

Until you can look at what switch is causing a problem.

 

Can you login to the switches config? As without unplugging anything might be able to see if it's a switch issue from dropping ports/connections etc

 

Steve

 

I agree - quickest (but most inconvenient) is to disconnect one switch at a time from your core switch (assuming you have one of course) whilst pinging a known problem IP. When the ping is stable you've found your dodgy switch and you can break it down from there.

[IAMCloud_Curt]

I agree - quickest (but most inconvenient) is to disconnect one switch at a time from your core switch (assuming you have one of course) whilst pinging a known problem IP. When the ping is stable you've found your dodgy switch and you can break it down from there.

 

Well if it is one, it's generally a bit of brute force. Depending how many switches etc you have in that side/floor.

 

Lets say you got 4 switches there. With one being the uplink.

 

Try connecting directly from uplink on laptop. Good pings?

Re-add one switch, and connect laptop to switch port. Good pings? etc etc

Until you can look at what switch is causing a problem.

 

Can you login to the switches config? As without unplugging anything might be able to see if it's a switch issue from dropping ports/connections etc

 

Steve

 

 

 

 

Only have 1 switch inside the cab, with 40 ports being used. Some are access points though. I'd probably say about 30 ports are workstations/printers so it won't be that difficult I assume?

 

Switch is Dell Power Connect 5548

PoE Switch is PowerDsine 3512G - I've already turned this off see eliminate this problem.

[Steve21]

If you have time, the easiest way, unplug everything.

 

Plug laptop in, and start ping test. One by one add a cable back, wait a minute and check ping. Repeat until one breaks it (But wait a least a minute so it can start looping/causing problems)

 

Steve

[IAMCloud_Curt]

Thanks mate, I can't do it right now because they are all using things, wouldn't want to disrupt them as they are already annoyed haha.

 

I'll be in here again tomorrow morning about 6:45ish so I can do it all then.

 

Will let you know the outcome mate, thank you.

[Steve21]

Thanks mate, I can't do it right now because they are all using things, wouldn't want to disrupt them as they are already annoyed haha.

 

I'll be in here again tomorrow morning about 6:45ish so I can do it all then.

 

Will let you know the outcome mate, thank you.

 

Only a personal opinion, but try to schedule it at lunch if you can (tell them you need 20 mins etc) as it could be something stupid that isn't plugged in/turned on in the morning, and it's only when XYZ does it it's an issue

 

Steve

[mikeyd101]

couple of things to look for/at the switch console /web/snmp and see what ports are generating most traffic. The other is break out something like wireshark and look at packet traffic, this may indicate bad nic in a device.

 

we've had a few loops here where people plug loose wires into the wall. A visual check might identify it.

[cpjitservices]

You might get away with running Wireshark in this instance, see if you can see anything flooding the network.

 

https://www.wireshark.org/

[IAMCloud_Curt]

Don't think it's a loopback, took everything out the other day.

 

Had 1 laptop plugged in next to me, pinged our gateway and still getting timeout's with nothing on the switch.

 

Faulty switch or config?

[strawberry]

Don't think it's a loopback, took everything out the other day.

 

Had 1 laptop plugged in next to me, pinged our gateway and still getting timeout's with nothing on the switch.

 

Faulty switch or config?

Or cabling..

[cpjitservices]

Have you tried plugging into Gateway Direct ? Pinging ?

[IAMCloud_Curt]

Have you tried plugging into Gateway Direct ? Pinging ?

Haven't tried that yet mate, will give it a go though. Thanks

[colacao82]

Have you got a fibre connection going to that switch? We had an issue like that here and one of our fibre cores was damaged.

[emdrum]

It would help if we had more information:

 

What's the IP address, netmask and default gateway of the devices you are pinging from and to?

Are they on different subnets? Is routing enabled?

Have you tried running traceroute to the device - if so, what result?

Are there multiple vlans? If so what is the vlan status of each of the ports between your PC and the printer?

How many other devices are on the same subnet? What kind of devices are they?

 

From what you said, it sounds like you have your printers running on the same subnet as your laptops and as your access points. My own guess is that you've got either a rogue device broadcasting rubbish or a broadcast storm on your network. If you watch the lights on the switches, are they flashing on and off or are they on pretty much all the time?

 

If it were a network loop then I think we'd see a much greater level of disruption. Spanning tree would be a good idea in any case, but it won't fix a broadcast storm.

[ReBoot]

In your original post you say half the school is experiencing issues with both your internal network and the internet.

 

This suggests an access switch or uplink is affected - are all the affected users on a specific access switch ?

 

If they are this should narrow down the problem space.

 

Have there been any changes to your network recently ?

 

One potential cause of intermittent packet loss is a duplex mismatch - one end is duplex the other half. This can occur when auto-negotiation is used and fails to setup the link correctly. (Usually best to setup duplex in the switch/router config where possible.)

[psydii]

I really would fire up wireshark on an affected workstation and see whether there were any clues in there.

 

Then I'd do the same from a port on the core switch, perhaps mirroring traffic heading for the internal interface of your firewall.

 

Almost every random network problem I have ever had has been either solved or at least better understood with a quick look at a wireshark capture.