Driftingforlife Posted March 24, 2017 Posted March 24, 2017 Morning everyone. I'm Rich, I'm the IT support guy at a school in gloucester and I really need some help. The network manager left on holiday yesterday (perfect timing) and I have never had the training for networking/advanced support. The server is on a Dell PowerEdge R810 running ESXI6. It has 4 active 1GB NICs. There are other VM's running off this host with no problems at all. Our 2008 R2 File server has ground to a complete halt. It started wednesday all of a sudden when the file system was responding slowly. When pinging the file server it was 100-500+ms. I restarted it and it cured it for the time being. Follow on yesterday where the same thing happend but even worse. When trying to copy/open a file it is very slow. Between 300kb/s - 1.25MB/s and during the ping time goes to 1000ms+. Also the system process & kernel time goes up from 20%-100% cpu usage. From much googleing I have uninstalled/reinstalled the VM NIC, disabled large Send Offload, disabled antivirus. I have been though the event logs and haven't found anything obvious apart from the time being out of sync which I corrected. No updates have been applied, nothing installed that I can tell. I have used the Windows performance tool kit to look at the system process to be honest I have no idea what I'm looking at. I was looking for any one thing with high useage but there seems to be several threads that are using the same. Anyone with an idea? I have 600 people looking at me to get this fixed and i'm stuck. Thank You Rich [TABLE=width: 217] [TR] [TD=class: xl157, width: 217]PowerEdge R810[/TD] [/TR] [/TABLE]
hallb15 Posted March 24, 2017 Posted March 24, 2017 My first thought is antivirus software - what AV do you use? Can you temporarily disable it and see if that helps.
mrwoberts Posted March 24, 2017 Posted March 24, 2017 (edited) Has this 2008 R2 FileServer had any updates applied recently, or any other changes, however unrelated... If the other VMs (using the same NICs) do not have a problem, could this be a NIC driver issue - is the Host using the specified driver from the Dell website? Ensure you've got an up-to-date backup of that VM before making any changes. Edited March 24, 2017 by mrwoberts
Driftingforlife Posted March 24, 2017 Author Posted March 24, 2017 (edited) My first thought is antivirus software - what AV do you use? Can you temporarily disable it and see if that helps. Sophos AV. I disabled it and tested and it made no difference. Has this 2008 R2 FileServer had any updates applied recently, or any other changes, however unrelated... If the other VMs (using the same NICs) do not have a problem, could this be a NIC driver issue - is the Host using the specified driver from the Dell website? Ensure you've got an up-to-date backup of that VM before making any changes. No updates since feb, I can't see any changes in the logs and i'm not aware of any being made since monday. I'm unsure if the driver came from dell. It has been working fine for months, just all of sudden this happened. It's also just started failing to apply GPs. Saying 0x800704cf The network location cannot be reached. Edited March 24, 2017 by Driftingforlife
mrwoberts Posted March 24, 2017 Posted March 24, 2017 Is that failing VM on a different disk/disk group to the other VMs?? I ask because I saw similar strange behaviour on a Dell T110 right before it decided to start rebuilding one of the disks in the array, which we swapped out to be on the safe side.
RobD Posted March 24, 2017 Posted March 24, 2017 Had a similar issue a while back and had to move the index file to a new drive to allow it to rebuild.
ollyyllo Posted March 24, 2017 Posted March 24, 2017 Are any of the other hosts using the same NIC as the problematic server? If so do they have a low ping?
Driftingforlife Posted March 24, 2017 Author Posted March 24, 2017 The OS is running on a RAID 10 array on the host. C has 17GB free out of 40GB. i'm not seeing any errors on the host through vSphere. The shared drives are on a HP SAN that is directly connected to the Dell server. The drives are showing up and working on the server but the drive maps on the server are not working again pointing to a NIC/driver/Kernel problem along with GP failing and sophos unable to update. Thanks guys for helping, no idea how much I appreciate it.
Driftingforlife Posted March 24, 2017 Author Posted March 24, 2017 Are any of the other hosts using the same NIC as the problematic server? If so do they have a low ping? We have 3 identical hosts. They are both working fine with 0msec ping. The SIMS server running on the same hose as the file server is also fine.
3s-gtech Posted March 24, 2017 Posted March 24, 2017 DNS issues around that server? Perhaps an IP address clash?
SpaceInvader83 Posted March 24, 2017 Posted March 24, 2017 After a reboot are pings back to normal? Pings from the server to other hosts normal or still high? Can you manually hit the network shares that the drives map to? Are these DFS shares?
SpaceInvader83 Posted March 24, 2017 Posted March 24, 2017 DNS issues around that server? Perhaps an IP address clash? Sounds a possible cause, power the server down and see if you can still ping the IP, I have had similar symptoms in the past.
Driftingforlife Posted March 24, 2017 Author Posted March 24, 2017 (edited) Tried several reboots with no luck. I just pinged the SIMS server on the same host which came back as <1ms. When I pinged the mail server on a differant host it came back with about 50ms....it's as soon as it passes through the host's NIC it becomes a problem....yet the sims server is fine. Will have to wait till later to be able to turn it off and test address. Going to get someone in to help as I'm over my head anyway. At Least it gives then a reason to send me on some extra courses haha. Thank you everyone for you help Edited March 24, 2017 by Driftingforlife
mrwoberts Posted March 24, 2017 Posted March 24, 2017 I would uninstall all Sophos components (temporarily) to rule out that little rascal - I'm no longer a fan of Sophos. Following a reboot, if the symptoms persists, update/rollback NIC driver. If you are still experiencing problems... grab yourself another cuppa, this is gonna be a long day ;-)
Driftingforlife Posted March 24, 2017 Author Posted March 24, 2017 Problem is now fixed. It was 1 of the 4 NIC's on the host. Was running at 100Mb not 1000Mb so its a dodgy cable/switch port/adapter. Disable it in VMware and all is well. Thank you everyone very much 3
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now