Intermittent Network

[LansITGuy]

Good afternoon.

 

I hope someone out there would be kind enough to help me. This post may drag on a little as I try and explain the situation. I don’t have much experience in solving networking issues like this so I’m not too sure what info is needed.

 

We currently have some quite major network connection issues in some of our rooms. This has been traced back to one of the cabinets which contains 4 switches. It appears that all ports in the cab are in some way affected. I have attempted to ping all 4 switches. The top 2 respond as normal but switches 3 and 4 have very intermittent ping responses with over 75% packet loss.

 

Switch 1 is connected to another router in my office and then connected to switch 2. Switch 2 is connected to switch 3, then switch 3 to switch 4. I have disconnected each cable from switch 3 in turn but the response rate does not appear to change. I have tested the connection cable between switch 2 and 3 which is working correctly.

 

At first I thought this was being caused by a feedback loop. However, the last time this happened in school the whole network was affected, but this time it is a much smaller area.

 

All switches are ProCurve 2626. I have logged into the web consoles and the software is reporting some errors (on these and other switches not listed above) including excessive jabbering and excessive broadcasts. There have been no network changes made in the past couple of weeks, but this problem only seems to have started last Friday.

 

Why is only a section of the network being effected? What can I do to trace and resolve this problem? Any suggestions would be very much appreciated.

 

Thanks for your time.

 

Mike.

[StewartKnight]

does sound like spanning tree protocol......

unless you have a device on the switch that is spamming the network

[FN-GM]

Have you made recent changes to the configuration to the switches?

[Gatt]

you haven't got a loopback somewhere have u? ie a network cable going from one wall port to another?

[Hybrid]

Try running WireShark or Ethereal on your network.

 

Also, which check which ports are being listed as sending out excessive broadcasts.

 

You could also pull one cable out at a time to find the offending device.

[LansITGuy]

Thanks for the replies. This just keeps getting stranger.

 

Whilst no one else was in school today I disabled all the ports with a view of starting them consecutively to see which port gave an issue. Even though the main PC ports were disabled I was still getting an intermittent ping loss when trying to contact the 3rd switch. I am really starting to think that it is the 2nd switch in the stack. To test this I bypassed switch 2 and so had 1 connected to 3.

 

This seems to resolve the issue (apart from leaving me 24 connections down). Thinking that this may have been a cable fault I reset the connection between 2 and 3 with a new cable. This seemed to clear the issue at first but now I’m back in the office it appears to have started again (18%) packet loss. As a point of interested I have tested the removed network cable and it is working correctly.

 

I really don’t think there is a loop back on the network. In the past this has stopped nearly all network activity and I’m just not seeing the same effects.

 

I will try getting hold of the WireShark and Ethereal tools to see what happens. It’s going to be a long day :? .

 

Thanks for the help guys. Any more suggestions are welcome.

[greenfieldsupport]

ive seen this horribleness twice here in greenfield.

 

The first time was an overheating 3com, try spacing your switches out abit more for cool-ness. We found we were getting problems mainly around 11am - 2pm, where we had the most traffic and generally the building / day was at its hottest.

 

The second time, which was almost impossible to detect, was an intel pro 1000 card on the server died. To the point where it was working, but not. The best way to check would be to use wireshark, if u got alot of black-ness (i.e. packets half formed or with invalid checksums) Its prollys the last problem.

 

Now what makes it hard to detect is that even when the server was powered off, or the cable removed, unless the switches were restarted the problem wouldnt go away!! and when the switches were restarted it would take a couple hours or even days before it would re-appear!!

 

so consider isolating the problem area, then turn off all the devices, and unplug them, then put just those two switches back on again with your laptop to monitor and perhaps one client at a time.

 

also remember that you may find that the problems have similar symptoms, if you find this fixes your problem it could be a bad NIC or it could be the switches are running alot cooler with the reduced load!

 

Sounds to me that you Definatly need to get a packet sniffer though, so you can see whats happening!.

Good luck!!!

[mattx]

I've been using Ethereal for a while - even when it supported Token-Ring from my old job.

Just installed and updated to wireshark and there is a huge improvement.

Anyway, I just took a 30 seconds sniff of our network and managed to capture someone trying to authenticate to a pop3 server - good thing is Wireshark showed their username and password !! HA HA HA.....

[LansITGuy]

I have run a few tests using the Wireshark program. Although I am not sure exactly what I would expect to find on a network loopback, the only errors I am seeing are coming from my own computer 8O . Using the Analyze – Expert Info tool I am getting a large number of Bad Checksum values. Just in case this was causing problems I have disconnected my own PC from the network, but the previously reported situation remains the same.

 

If anyone has a spare second I have attached a 1 minute sample output file. For reference the computer running Wireshark is on IP 172.16.1.9.

 

Thanks again everyone for your continued help.

hc_1min_sample.rar

[mattx]

Just had a quick look and you have quite alot of ARP requests....

[mattx]

Also - there are ARP requests for 10 addresses, 198 addresses, and 212 addresses !!

Just what is your IP range on your network / networks ?

 

I am NO expert, but I would be inclined to track down these devices first and turn them off, then see what happens.

[LansITGuy]

Ok, the 10 range is our Admin network which has a domain trust with the curriculum side. The 198 and 212 ranges? Good question.... Time to start digging.

[mattx]

Sorry, just to ask too - [ and i may be wrong but what is device 192.168.0.1 ? Is it a D-link box of some sorts ?

The reason I ask is that sometimes that is used as a default address for an AP - and that may be trying to hand out IP address...

I take it you are running DHCP ? Is your DHCP server 172.16.1.168 by any chance ?

Like I said I am no expert in these matters !!

 

Scrap the DHCP theory, 192.168.0.1 is sending out SSDP...

[LansITGuy]

Thanks for the help Matt.

 

Ok, the DHCP for the network is 172.16.0.1. I think there may have been a D-Link router used to add network ports in our staff room. Just going to head off to power it down. Hopefully we may be on to something.

 

Cheers.

[mattx]

No problem, I hope you get it sorted, I am not so sure now on the DHCP front - looking further down the trace and sorting it via IP address. Still I may be wrong again !!

Will be interesting to find out what is 192.168.0.1 though......[ and what its doing ]

 

I'm not around this PM - off to my sons school for a bit - but let us know if you manage to find anything !! [ I'll check later this evening ]

[mattx]

I'm a little confused as to this 192.168.0.1 device as its basically telling another device it's DHCP address is either wrong or its moved to another subnet and with you stating your DHCP server is 172.16.0.1 ..... well....

[LansITGuy]

Turned off the router in the staff room with no change in ping resposnes to the effected routers. The strange thing is that users logins are not being slowed down to the same level they were yesterday.

 

I am fairly sure that there is some problem somewhere with the link between my office cab and the one containing the effected router. There are excessive communication errors being logged in the routers consoles passing down the chain of connected ports 25 & 26, but it is only the 3rd router in the cab which has the intermittant communication issues.

 

The switches are quite well spaced and ventilated so I don't think we have an overheating issue. I may trying rewiring the cab round as the 4th switch doesn't seem to have any active connections and so I can move all of switch 3 cables into it to see what happens.

[greenfieldsupport]

Okay quick breif update, i'll look in more depth but you dont have a problem like mine, with the network card.

 

The very little black is due to your card attempting to do checksum offloading, where the packet is captured before it recieves its correct checksum and so shows up invalid.

 

You dont have to worry about this at all..

[greenfieldsupport]

Seems fine, abit of a noisey environment.

 

could you capture a couple more minutes, and when you do could you type in your terminal ping 'problem switch ip address' -t -l 1200

 

192.168.0.1 is setup as a DHCP server, you should disable that if its not ment to be.

 

You can see this as when your clients ask if their ip address is current and okay, it replys saying that the lease has expired / didnt exsist / is wrong (NAK)

 

It looks like your clients arnt paying any attention to its NAK's as they dont then ask for a new IP address. But you should still disable DHCP on it unless you have a reason to leave it on.

[torledo]

Turned off the router in the staff room with no change in ping resposnes to the effected routers. The strange thing is that users logins are not being slowed down to the same level they were yesterday.

 

I am fairly sure that there is some problem somewhere with the link between my office cab and the one containing the effected router. There are excessive communication errors being logged in the routers consoles passing down the chain of connected ports 25 & 26, but it is only the 3rd router in the cab which has the intermittant communication issues.

 

The switches are quite well spaced and ventilated so I don't think we have an overheating issue. I may trying rewiring the cab round as the 4th switch doesn't seem to have any active connections and so I can move all of switch 3 cables into it to see what happens.

 

I'm sorry but I find it really hard to visualize all or part of a network without a schematic drawing or diagram of some sort, Switch A connected to Switch B connected to Router Z doesn't really help me to understand your network layout - at first I assumed you've got a problem with a switch stack and now you're mentioning a router issue ?

 

As you know the first step in diagnosing any problem is looking at the layer 1 layout for this section of the network and how it connects to your core.

 

Some possible cause;

 

If you're Procurves are connected in a stack it could be a defective switch.

 

As somone has suggested it could be an STP loop possibly caused by a defective link(s). Or the stack interswitch links

 

Could be a broadcast storm - try and isolate the culprit device/devices or link(s) that are causing the problem.

 

Putting a packet trace on the wire isn't a bad idea but normally you can determine switching issues from the messages spewing from your switches - check and double check your physical links (copper and fibre) and your switch config files drilling down port by port. Google the notificatin messages coming from your Procurves.

 

Check any Etherchannel or trunk links you've got setup are working correctly. You may not have changed the config on your switches, but if you've had a power cut or someone has manhandled your uplink cabling then something beyond your control has been altered. Finally be methodical about checking the hosts connected to the suspect switches.

[LansITGuy]

Thanks again guys for all the continued suggestions.

 

Sorry about the lack of a network diagram. There hasn't been one in school and it wasn't until quite recently that I got to understand this side of the system (I am mostly here for software support & web dev).

 

Regarding the issues of the staff room router, this was not the problem. Disconnected it completely to see the same ping responses.

 

I am fairly sure I have narrowed it down to a single switch as when I bypass this piece of hardware all network responses return to normal. I think it is now a case of going through the ports one by one to see whether it is a hardware issue or an item attached to one of the ports.

 

I am still in the process of trying to trace the other machine attempting to be DHCP.

[gaz350]

could a rogue machine have Windows Internet Sharing turned on?

[greenfieldsupport]

You can see from the capture its not a broadcast storm or a loopback.

plus loopbacks are easy to spot as your activity lights will flash like mad with no pause

 

 

If i was you Id replace the suspect with a known good switch and test for a few days. If you still have problems, dont be lazy and REMOVE the cables form the ports, disabling them may still allow an electronic defect to disrupt your network (like when i had that bad network card)

 

to test just put ping "Ip" -t

and leave it running, at the end of the day press ctrl + C and it will give you the summary.

look for amount of lossed packets, even a 2% loss is bad enough, anything higher is terrible!