School Networking Issue - Random disconnecting machines - Plea for help :)

[tj2419]

We are currently experiencing a strange issue on our network and I am hoping someone might be able to give me some suggestions ☺.

 

What we run: Windows Server 2003, CC4, XP workstations, two networks an admin and a curriculum. The admin network is vanilla.

 

We are getting random computers disconnecting from our network. Sometimes single machines other times whole classrooms or 4 classrooms at a time. This has been happening since September sporadically. Over the summer we had a new CCTV system (avermedia) installed, two backup servers using backup exec and a new HP switch connecting one of the backup servers at the other end of the school. We did also set up a separate OU on one of the curriculum servers and set up a set of windows laptops. But this was set up 6 months before the issues started happening although we did make on change to a group policy allowing for the users my documents to be mapped.

 

When machines disconnect they loose their IP address and DNS servers.

 

These disconnections happen on both our admin and curriculum networks.

 

Symptoms.

Computers will loose network connection (get the yellow warning triangle in the system tray saying “no or weak signal”). The users loose access to the internet and my documents, shared drives and database software like access fails as it has lost connection. To gain connection again some just require unplugging the Ethernet lead and plugging it back in. Others (when whole rooms go) require the switches they are connected to be restarted.

 

When the machines disconnect the error logs just show that the RM service host failed. But I imagine this is as a result of the disconnection. I am leaning against an RM issue as it is happening on our vanilla admin network.

 

On the core switch we get errors building up on random ports that when left on the original core switch went up into their thousands in a few weeks.

 

More recently (February onwards)we have started having Group policy issues where computers are storing the username of the last user in the login box, some machines not getting software shortcuts until GP update is run on the machine. Mainly this is isolated in the one classroom.

 

We also recently has an issue where one morning we came in to find the RM service host had crashed and wouldn’t restart. On about the second restart of all the servers it kicked in.

 

Attempted solutions.

• Stopped backup exec services on all servers and unplugged the backup servers to check backup traffic wasn’t over loading the servers.

• Unplugged CCTV box to check that wasn’t causing issues.

• Restarted all switches.

• Replaced Core switch – when we replaced the core switch it did seem better for a week or so and then started again.

• Changed stacking kit in main cabinet so each switch had 1GB Ethernet connections.

 

 

Sorry if this makes no sense but I thought if I put all the information out there it might trigger something with someone.

 

Any ideas?

 

If the issues are on both admin and curriculum networks would this point to switches being at fault or could it still be a CC4 issue?

 

Any help / advice would be great thanks as I am out of ideas.

 

Cheers

[glennda]

Random question - do you use securus? is it up to date?

 

it tends to do annoying strange random acts of stuff like this

[tj2419]

Random question - do you use securus? is it up to date?

 

it tends to do annoying strange random acts of stuff like this

 

Nope not a securus user. Thanks for the reply

[Michael]

This does sound like a switch(es) problem to me. I'm guessing you have other switches plugged into the core switch?

 

From past experience, I've found actually turning a problematic switch off, unplugging it for upto 60 seconds and plugging back in works better than simply restarting.

[deano]

Definitely a switch issue to me. Sounds like a broadcast. Do you have spanning tree enabled on your switches?

[tj2419]

This does sound like a switch(es) problem to me. I'm guessing you have other switches plugged into the core switch?

 

From past experience, I've found actually turning a problematic switch off, unplugging it for upto 60 seconds and plugging back in works better than simply restarting.

 

When I reset thats what we do. The switches are managed by the council so I only have access to the logs on the core switch nothing else at the moment.

 

Are there any other likely causes if it's not the switches? Or would you guys say pretty confidently it's the switches?

 

Thanks

[Michael]

I'd stick my neck out and say (from the info you've given) and the fact you've replaced the core switch, I'm 99.9999999999% confident it's a switch issue.

 

At the end of the day, if the switches are managed by the council you should definitely make them aware of your issues and get them to dial in.

[tj2419]

They are. So far they have been trying for over 6 months to fix it!! It took me 2 months to get them to agree to swapping the core switch to see if that could be the issue. I am wondering if possible there is a clash from the firmware of the new switch to the other switches that is causing the issues. Could that be a possible cause?

 

Got to love these forums thanks for all the comments guys

[CyberNerd]

have you got vlans or any routing set up on them?

[tj2419]

have you got vlans or any routing set up on them?

 

Sorry my knowledge of switches is fairly limited. But from a quick google i don't think we do have any VLANs.

 

Cheers

[CyberNerd]

Sorry my knowledge of switches is fairly limited. But from a quick google i don't think we do have any VLANs.

 

Cheers

 

 

ok. what exactly are the errors that you see on the core?

How big is the network (number of switches)

can you log into the switch and ping a host, see the port the mac address is coming from?

 

They are. So far they have been trying for over 6 months to fix it!!

 

alternatively can you sack them and get a consultant in?

[tj2419]

This is a copy of the log from one of the ports with errors. I do have to say since changing the core switch the number of errors has dramatically decreased.

port: GigabitEthernet1/0/2

 

Peak value of input: 5001134 bytes/sec, at 2000-06-06 23:57:40

Peak value of output: 4842600 bytes/sec, at 2000-05-05 04:07:14

Last 300 seconds input: 93 packets/sec 10673 bytes/sec 0%

Last 300 seconds output: 97 packets/sec 36362 bytes/sec 0%

Input (total): 940295537 packets, 318311186090 bytes

937088408 unicasts, 2242302 broadcasts, 964780 multicasts

Input (normal): 940295490 packets, - bytes

937088408 unicasts, 2242302 broadcasts, 964780 multicasts

Input: 47 input errors, 0 runts, 47 giants, 0 throttles

0 CRC, 0 frame, - overruns, 0 aborts

- ignored, - parity errors

Output (total): 759778827 packets, 399820918224 bytes

734846446 unicasts, 17582701 broadcasts, 7349680 multicasts, 0 pauses

Output (normal): 759778827 packets, - bytes

734846446 unicasts, 17582701 broadcasts, 7349680 multicasts, 0 pauses

Output: 0 output errors, - underruns, - buffer failures

0 aborts, 0 deferred, 0 collisions, 0 late collisions

0 lost carrier, - no carrier

----------------------------------------------------------

 

The network is in the region of 20-25 switches.

 

"can you log into the switch and ping a host, see the port the mac address is coming from?"

Sorry how would i do this?

 

Thanks

[User3204]

If there are no VLANs on the switches, then is there any other configuration on them ?

You coult try wiping the config on them, it's usually something simple, I know with the HPs you just boot it up with port 1 connected to port 2.

 

We had an issue with our switches, it was only affecting our phones on their own VLAN, it turned out to be the fact that one server had DHCP Snooping switched on, the problems went away when I switched it off.

 

Do you have a single flat network ? If so, then perhaps separating what you can may help to narrow down the problem.

How regular is the problem ? If you get the same thing every day, then try disconnecting one whole ICT suite (switch off the switch for this suite, and tell people it's "Off"), as this may also show if it is some devices in a specific room.

 

 

years ago, when we used to have loops, usually due to some git plugging two sockets into eachother, when the loop would kill my network, I would just randomly unplug every single cabinet until the network calmed down, we could see the high-traffic. But I do have 12 distribution cabinets, so pulling out fibres until the packets stop is very easy to watch.

 

 

 

Have you got a hold of Wireshark ? Stick it on the DNS/DHCP server, and have a look through.

[User3204]

Peak value of input: 5001134 bytes/sec, at 2000-06-06 23:57:40

Peak value of output: 4842600 bytes/sec, at 2000-05-05 04:07:14

 

Aww man, the date is wrong on those switches, this used to wind me up on my old HP, Cisco ones, now that I have a standard switch make across the site, it was so much easier to make them all sync from my NTP server.

 

 

 

What brand switch(es) do you have ?

How old is the oldest ?

[tj2419]

If there are no VLANs on the switches, then is there any other configuration on them ?

You coult try wiping the config on them, it's usually something simple, I know with the HPs you just boot it up with port 1 connected to port 2.

 

We had an issue with our switches, it was only affecting our phones on their own VLAN, it turned out to be the fact that one server had DHCP Snooping switched on, the problems went away when I switched it off.

 

Do you have a single flat network ? If so, then perhaps separating what you can may help to narrow down the problem.

How regular is the problem ? If you get the same thing every day, then try disconnecting one whole ICT suite (switch off the switch for this suite, and tell people it's "Off"), as this may also show if it is some devices in a specific room.

 

 

years ago, when we used to have loops, usually due to some git plugging two sockets into eachother, when the loop would kill my network, I would just randomly unplug every single cabinet until the network calmed down, we could see the high-traffic. But I do have 12 distribution cabinets, so pulling out fibres until the packets stop is very easy to watch.

 

 

 

Have you got a hold of Wireshark ? Stick it on the DNS/DHCP server, and have a look through.

 

We do have a mitel IP phone system that maybe on a VLAN. We have never managed the phones so i will check on that tomorrow. If it is how could a VLAN be causing the issues?

 

We had a similar issue a few years back when i came to the school and the previous senior technician had used ethernet splitters in classrooms which caused havoc.

 

I'll have a look at wireshark tomorrow as well.

[tj2419]

Aww man, the date is wrong on those switches, this used to wind me up on my old HP, Cisco ones, now that I have a standard switch make across the site, it was so much easier to make them all sync from my NTP server.

 

 

 

What brand switch(es) do you have ?

How old is the oldest ?

 

They are 3com switches. Been here longer than i have and when i asked the council they couldn't give me an answer :|

[tj2419]

Is there anything specific to look out for in wireshark?

[CyberNerd]

They are 3com switches. Been here longer than i have and when i asked the council they couldn't give me an answer :|

Not sure about the older menu driven comware versions, but newer comware is quite straightforward.

what model?

 

We do have a mitel IP phone system that maybe on a VLAN. We have never managed the phones so i will check on that tomorrow. If it is how could a VLAN be causing the issues?

 

There could be a loop between the different vlans caused by a misconfiguration. Check with wireshark to see if you see any VoIP broadcasts on the other network.

[nathan3388]

Have you checked that you have not got a loop through on the network. this will broadcast packets and sometimes lose connectivity we had this problems as we didn't have spamming tree turned on

[tj2419]

Have you checked that you have not got a loop through on the network. this will broadcast packets and sometimes lose connectivity we had this problems as we didn't have spamming tree turned on

 

How would we check for a loop through on the network? Spanning tree, is this turned on, on the core switch? Any draw backs from enabling it? Cheers

[glennda]

How would we check for a loop through on the network? Spanning tree, is this turned on, on the core switch? Any draw backs from enabling it? Cheers

 

no - should be enabled on all - As you ignored me previous post I guess you don't use securus?

[nathan3388]

go around and check cabling we had a kid plug one network cable in one port then pluged it into another slowed the network right down

[tj2419]

no - should be enabled on all - As you ignored me previous post I guess you don't use securus?

 

Hi I replied to your post :s the third post on here :s but we don't use securus. I'll have a look at the switches this morning.

[meastaugh1]

If it were a loop that hadn't been disabled by STP, wouldn't you expect slightly different symptoms such as performance degradation across the entire LAN as opposed to the sporadic loss of DHCP lease info?

 

Have any clients with static IPs ever been affected?

 

I've seen similar issues when a dulicate DHCP server was introduced to the network in the form of an AIO TFTP/DHCP/DNS utility. If you haven't yet done so, it's worth having a look at event log on your DHCP server and perhaps sniffing for evidence of a duplicate DHCP server.

Edited June 13, 2012 by meastaugh1

[davewm]

Microsoft Network Monitor - slightly easier to use than wireshark and not the same amount of overhead installing it - I've shifted to using it rather than wireshark.

 

I've seen similar behaviour from a router / core switch / managed wireless controller having the same IP as another device on the network.

 

If you have a CCTV / IPTV system I would strongly suggest that this is put on its own VLAN - your IP phones should be on their own VLAN too.