Network issues with network over last week.

[IT_Man_Dan]

Good Morning all,

 

I've got an issue with my network that is driving me crazy...

 

The symptoms are:

Remote desktop to our main server keeps dropping connection (every minute or so!).

Mapped drives temporarily loose connection.

Shared start menus and icons randomly drop out (pressing F% sometimes brings them back).

 

This has only been happening over the last week, before then all was fine. Nothing to my knowledge settings wise has been changed. I did/do suspect it to be DNS related but have had another IT specialist from county look at all the settings on Friday and he says they are ok?!

 

I have tried swapping to a different NIC to rule that out but the problem persists.

 

Does anyone have any ideas of what else it could be please?

 

Thanks in anticipation.

[synaesthesia]

Can you check the logs of the switches and ping from other machines to other machines OK?

 

When pinging over a while (using ping -t) do you get dropouts?

If so, and it's quite random (i.e. there's no solid set of pings for 20 secs at a time) it would sound like a network loopback killing a switch. Can get the same symptoms from faulty network hardware too including switches themselves, wifi APs, bad nics etc.

 

If that's the case, isolate each switch until it pings OK (or remove all then plug in one by one until it stops pinging properly) - obviously making sure the machines you're pinging are still physically connected

Then go hunt for the cause.

[mrbios]

I'd say start off with the most basic of changes first by changing network cables. Also check for any loops. Sometimes the most irritating of problems can end up being the simplest of fixes

 

Then work your way on to the switches configuration, before eventually moving on to the servers.

[cpjitservices]

What switches do you have ?

Have you got spanning-tree enabled ?

and also could be a faulty cable, could be a faulty cable from the PC to the wall or from the wall to the patchpanel and from the patch panel to the switch. You might have to trace your cables and try and change them see if the error still occurs.

[seawolf]

Good Morning all,

 

I've got an issue with my network that is driving me crazy...

 

The symptoms are:

Remote desktop to our main server keeps dropping connection (every minute or so!).

Mapped drives temporarily loose connection.

Shared start menus and icons randomly drop out (pressing F% sometimes brings them back).

 

Thanks in anticipation.

 

I assume this is happening with all clients on the network? If it is, then you need to be checking the switch logs to see if you are experiencing a lot of collisions or flooding of the network from specific network ports. You could have a loop somewhere, but if you have STP enabled that's unlikely.

 

I would be trying to isolate it to a particular switch or ports(s) because you might have a faulty switch, NIC, or a compromised client(s) flooding the network (malware or botnet). You might also see if you're switch firmware is up-to-date. We had some network problems earlier this year out of the blue that was resolved by a firmware upgrade to all of our switches.

[IT_Man_Dan]

Thanks for your replies so far guys, I'm going to start checking the switches now. The problems do seem to be network wide but the reason I believe the fault lies with the server is that when you remote onto any other server the connection doesn't drop. It's only when you remote on to the pdc (where the drives, start menus, and desktops are hosted) that the connection randomly drops out?! Drives etc on other services are unaffected. I've tried different cable and NIC on the pdc.

 

Thanks for all your support so far.

[cpjitservices]

I've tried different cable and NIC on the pdc.

 

 

Sounds like it could be the connection back to the switch, or maybe a duplex negotiation problem ? Are you running gigabit or 10/100 ?

[IT_Man_Dan]

Ok I have now gone into all of my switches (All HP procurves) and checked them. They all seem ok apart from four that didn't have spanning tree enabled, I have now enabled it on those four switches. I've also found one with excessive CRC/Allignment errors on a port, I plan to go into that room in a few minutes to check cabling and take that switch offline to see if it has any effect, It was one of the switches without spanning tree enabled.

 

I'm still struggling to come to terms with the fact that it's cable/switch based. It only seems to be effecting connections to one server and I have tried a different NIC and cable to it.

I have restarted that server this morning and still have the problems.

 

Thanks for your ongoing support.

[IT_Man_Dan]

Ok, I'm back looking at the server event viewer to try and figure out the issue.

 

One thing I have noticed is that I'm getting a lot of SRV warnings Event ID:2012 this does sometimes coincide with my remote session dropping to the server...

 

Could this be it?

[CyberNerd]

log into the switch that is connected to the server.

from the switch, ping the server's IP address. Note the mac address of the server using ARP.

When the server drops ping, does the mac address change?

[seawolf]

Ok, I'm back looking at the server event viewer to try and figure out the issue.

 

One thing I have noticed is that I'm getting a lot of SRV warnings Event ID:2012 this does sometimes coincide with my remote session dropping to the server...

 

Could this be it?

 

The 2012 errors would tend to indicate a NIC problem or an issue with the switch the server is attached to. Is the switch with the CRC errors the same the server is connected to? Have you tried to upgrade the NIC drivers?

 

What HP switches do you have? I had an issue with 2520g-24 switches that were suddenly causing a lot of problems until firmware was upgraded. Just be aware the very latest HP firmware for some switches has a totally different (non-Java) GUI, s if you rely on the switch GUI proceed with caution.

[IT_Man_Dan]

The HP switches are: mainly procurve 2650's with a procurve 4108GL (server room, one server is connected to), a couple of 2626's and a couple of 2610's.

 

I can ping the server itself from a workstation and it won't drop a ping, even when I loose my icons and mapped drives or get kicked off a remote connection to the server it can ping it?!?

[DMcCoy]

Duplicate IP?

[seawolf]

The HP switches are: mainly procurve 2650's with a procurve 4108GL (server room, one server is connected to), a couple of 2626's and a couple of 2610's.

 

I can ping the server itself from a workstation and it won't drop a ping, even when I loose my icons and mapped drives or get kicked off a remote connection to the server it can ping it?!?

 

I would take a look at the firmware that is on the 4108GL. We have a 4208VL that we also had to upgrade the firmware on a few months ago due to persistent CRC/Alignment errors. The issue was resolved with newer firmware.

[IT_Man_Dan]

Thanks for all your replies so far.

I will look into upgrading the firmware on the HP 4108GL tomorrow.

 

After reading other threads and doing other research I'm wondering if I should really consider teaming the NIC's (something I have previously considered)?! The box is a Dell T410 with onboard twin Broadcom NIC's and an extra twin NIC (intel I believe). Currently it is only using 1 Broadcom NIC out of 4 into the HP 4108GL so I'm wondering if getting the other Broadcom NIC teamed with it would help?

What are your thoughts on this?

 

Thanks

[CyberNerd]

Great idea but you might want to fix the problems first.

[laserblazer]

We have similar symptoms but more intermittent. There are currently two nics that are teamed. Tomorrow, as part of the elimination process, we will try just one nic and see what happens.

[IT_Man_Dan]

I just wanted to update this thread with the current situation.

 

I haven't resolved the issue but I have got a lot less srv errors on the server. From looking at the logs the srv errors only occur during the day when the network is in use suggesting it's switch/bandwidth issues or a certain piece of hardware. The last srv error we had was at 12:53 yesterday afternoon and we appear to be getting less and less *touch wood*. Unfortunately I'm still experiencing users loosing their redirected desktops, start menus, or temporary access to mapped drives and it's driving me crazy. All these are redirected from the main server which was having the srv errors and dropping remote sessions intermittently.

 

What did seem to slightly relieve the issue was to plug the server into a different port on our 4108gl switch which is our main switch in our server room, since then the srv errors have decreased but the icons still go missing etc. I suspect that it might be this switch causing the issues. One thing I forgot to mention is that we had a long power cut Wednesday night and all our servers, etc shut down.

I haven't as yet upgraded the firmware on the switch as I'm very nervous about doing so as it is our main switch and I don't currently have a spare. Another thing I'm considering doing is teaming the broadcom NIC's in the server to spread the bandwidth (now that I'm getting less srv errors).

 

Could it be bandwidth related that users intermittently loose their icons, etc? It does suggest so as the busier the network gets the more the problem shows itself, all was fine until last week though?!?

 

Another thing I've done is upgraded Sophos on all of our clients to 10.2 from 10.0 yesterday morning (not that I think this will effect anything but thought I should let you know).

 

Further feedback and advice is much appreciated.

 

Thanks

[chazzy2501]

Have you tried disabling some of the hardware accelerations on the NIC? (flow control, arp offloads, tcp offloads) I find this helps if the driver is funny. (some of those options mess up tcpip)

[cpjitservices]

Are you using VLANS ?

 

You could separate all of your servers, pc's, phones into VLANs and see if that improves, segregate your broadcast domains because if your not using VLANs your entire switch is a broadcast domain and it could be slowing your network. Especially if you have a broadcast storm. Sounds to me actually like you *might* have a switch loop somewhere, watch out for the port LED's on your switch flashing really fast or more than usual, especially if they all do it at the same time.

 

A bit like this:

Edited May 1, 2013 by cpjitservices

[seawolf]

I just wanted to update this thread with the current situation.

 

Further feedback and advice is much appreciated.

 

Thanks

 

Have you used Wireshark to see what is going on at the time of the SRV errors and other issues? You might also want to look at using MRTG or PRTG to check the bandwidth on all of the ports (or download and install OpManager, you get full features for the trial period).

 

Also, is this server a VM or a physical server? I have seen similar issues occurring when a server was being pushed beyond its capacity, particularly in regards to disk I/O. If you are seeing high read or write latencies, then you will want to look into that further. This will cause some big problems with VM servers in particular.

[IT_Man_Dan]

The server is a physical one.

 

My NIC advanced settings are listed below, please can you advise any changes to them that I should make?

 

Broadcom BCM5716C NetXtreme 2 GigE (NDIS VBD Client)

 

Ethernet@wirespeed = Enabled

Flow Control = Disable

Interrupt moderation = Enable

IPV4 Checksum Offload = TX/RX Enable

IPV4 Large send offload = Enable

IpV6 Checksum = TX/RX Enable

IPV6 Large Send Offload = Enable

Jumbo MTU = 1500

Locally Administered Address = Not Present

Number of RSS queues = 8

Priority and VLAN = Priority and VLAN Enabled

Receive Buffers = 750

Receive Side Scaling = Enable

Speed & Duplex = Auto

Transmit Buffers = 1500

VLAN ID = 0

 

Thanks for your help.

[seawolf]

The server is a physical one.

 

My NIC advanced settings are listed below, please can you advise any changes to them that I should make?

 

Broadcom BCM5716C NetXtreme 2 GigE (NDIS VBD Client)

 

Ethernet@wirespeed = Enabled

Flow Control = Disable

Interrupt moderation = Enable

IPV4 Checksum Offload = TX/RX Enable

IPV4 Large send offload = Enable

IpV6 Checksum = TX/RX Enable

IPV6 Large Send Offload = Enable

Jumbo MTU = 1500

Locally Administered Address = Not Present

Number of RSS queues = 8

Priority and VLAN = Priority and VLAN Enabled

Receive Buffers = 750

Receive Side Scaling = Enable

Speed & Duplex = Auto

Transmit Buffers = 1500

VLAN ID = 0

 

Thanks for your help.

 

What's the MTU seton the switch port the server is connected to? If it doesn't match the server, then that spells trouble. Also, I would be more inclined to leave flow control turned on at the NIC and disable it on the switch ports (all of them). However, if its off both on ALL of your NICs OR off on ALL switch ports then that won't be the cause of any problems. The sudden occurrence of your problem would seem to rule out flow control being an issue unless you have recently added any new devices (NICs) to the network that would correlate to the timing of your problems.

[IT_Man_Dan]

Argghhh 10 srv errors within the last hour and all icons etc keep disappearing intermittently..

[chazzy2501]

well as flow control is off turn it on. This should stop packet loss due to high traffic. (This is believe is already built into the tcp stack so you shouldn't need it)