Event ID 2012 Source Srv on Server 2008

[dhoward_westexetc]

Hey All,

 

Has anyone got an updates on this issue? Just noticed it on my hyper V file server...

 

Ross

 

Nothing definite yet but I have some idea of what is causing it, in our case it may be due to the setting in McAfee VirusScan 'block the connection when a threat is detected in shared folder' that may be interfering with SMB on the 2008 servers. I think its only consequence is occasional latency issues, I had originally thought it was causing issues with loss of data in Word 2003 on save but I think now the two are unconnected, and that Word itself is the cause of that. I'm waiting until all of our McAfee clients are 8.7i before I really see for certain if it has resolved it.

 

Take special note on whether the Srv 2012 errors are happening in the holidays. In our case they are not, indicating that it is an error originating from clients.

 

From looking just now, I think the number of errors is starting to decline slowly after we replaced most of our 8.5i installs with 8.7i. Performance is improving all the time.

[januttall]

hi we seem to be having a simaler issue on boath of our 2008 servers one is enterprise 2008 and the other is the R2 version we also have a 2003 running in the background for AV etc that does not display the errors. in our case it seems to cause a large number of winlogon errors and even cause machines to reboot we have turned off chimeneying and upgraded the drivers and installed all of the relivent upgrades and paches what is strange is that the problem gets worse the further from the servers you go. we have tested the network as much as we can and even run packet captures to try and find a solution. but we think the problem is server based as it moves from machine to machine any help or advice would be greatly apreciated. we are using sophos antivirus and all of our clients are currently XP SP3 Edited January 17, 2011 by januttall

[itgeek2008]

I am now getting this issue, but it has been only since I installed service pack 2 on the servers

[januttall]

we have found that our main domain controler was spitting out packets with ilegal checksums and curupted data we turned of chimney and tcp ofloading and it has scince stoped transmitting bad packets this has helped but we are still seing machines rebooting with error codes 1004,1005 as well as the following error dumps.

 

Microsoft ® Windows Debugger Version 6.12.0002.633 X86

Copyright © Microsoft Corporation. All rights reserved.

 

 

Loading Dump File [C:\technician\errors domain\client\Bginfo.exe.hdmp]

User Mini Dump File: Only registers, stack and portions of memory are available

 

Symbol search path is: *** Invalid ***

************************************************** **************************

* Symbol loading may be unreliable without a symbol search path. *

* Use .symfix to have the debugger choose a symbol path. *

* After setting your symbol path, use .reload to refresh symbol locations. *

************************************************** **************************

Executable search path is:

Windows XP Version 2600 (Service Pack 3) UP Free x86 compatible

Product: WinNt, suite: SingleUserTS

Machine Name:

Debug session time: Wed Dec 1 14:47:07.000 2010 (UTC + 0:00)

System Uptime: not available

Process Uptime: 0 days 0:00:04.000

.................................................. ....

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(c70.c74): Access violation - code c0000005 (first/second chance not available)

eax=c0000005 ebx=80070000 ecx=0012bb98 edx=00000000 esi=000005a0 edi=00000000

eip=7c90e514 esp=00129a28 ebp=00129a8c iopl=0 nv up ei ng nz ac pe cy

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297

Unable to load image C:\WINDOWS\system32\ntdll.dll, Win32 error 0n2

*** WARNING: Unable to verify timestamp for ntdll.dll

*** ERROR: Module load completed but symbols could not be loaded for ntdll.dll

ntdll+0xe514:

7c90e514 c3 ret

 

and

 

HI

 

This is the second dump file

 

Microsoft ® Windows Debugger Version 6.12.0002.633 X86

Copyright © Microsoft Corporation. All rights reserved.

 

 

Loading Dump File [C:\technician\errors domain\client\winlogon.exe.20110124-112714-00.hdmp]

User Mini Dump File: Only registers, stack and portions of memory are available

 

Symbol search path is: *** Invalid ***

************************************************** **************************

* Symbol loading may be unreliable without a symbol search path. *

* Use .symfix to have the debugger choose a symbol path. *

* After setting your symbol path, use .reload to refresh symbol locations. *

************************************************** **************************

Executable search path is:

Windows XP Version 2600 (Service Pack 3) UP Free x86 compatible

Product: WinNt, suite: SingleUserTS

Machine Name:

Debug session time: Mon Jan 24 11:27:17.000 2011 (UTC + 0:00)

System Uptime: not available

Process Uptime: 0 days 1:10:55.000

.................................................. ..............

.......................................

Loading unloaded module list

................

This dump file has an exception of interest stored in it.

The stored exception information can be accessed via .ecxr.

(2a4.2a8): Access violation - code c0000005 (first/second chance not available)

eax=c0000005 ebx=80070000 ecx=00069898 edx=00000000 esi=000009fc edi=00000000

eip=7c90e514 esp=00067728 ebp=0006778c iopl=0 nv up ei ng nz ac pe cy

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000297

*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll -

ntdll!KiFastSystemCallRet:

7c90e514 c3 ret

[dhoward_westexetc]

A quick update on this from my end:

 

I updated all our client McAfee installs to 8.7i and I am still getting the errors.

 

However for our servers there seem to be a barrel load of updates recently for NIC, BIOS and RAID, which I plan an update-a-rama at Easter, so I will see if this resolves the issue.

I'll try the suggestion by januttal as well to see if that helps, but it only happens on file read/write in our case.

[dhoward_westexetc]

My latest update:

 

I applied all of the NIC and RAID firmware updates and drivers to our 2008/R2 servers. I've upgraded the servers to McAfee 8.8i as well.

 

Network performance is like night and day, all of the new servers are so responsive it is scary! Our McAfee local mirror on our 2008 R2 server works consistently for the first time ever! Since those upgrades, auto-tune receive window is also now working as I expect, with transfer speeds of around 60 MB/s, which is what I expect between a 2008 SP2 server and a Windows 7 client.

 

We will only know next week if that fixes the srv 2012 errors, but things seem more reliable and responsive already. If that is the case then I can only conclude it is a malfunctioning network firmware/driver that has been corrected with those updates. It would tally with what januttal says in terms of bad packets being produced.

[bulldog1]

dhoward how did your updates fair? Did it resolve the issue. I found this thread and I'm experiencing the same issue on a Windows 2008 R2 File server. No McAfee just Forefront installed.

[dhoward_westexetc]

dhoward how did your updates fair? Did it resolve the issue. I found this thread and I'm experiencing the same issue on a Windows 2008 R2 File server. No McAfee just Forefront installed.

 

Didn't resolve the issue i'm afraid. I'm guessing its a client generated issue therefore, since our servers now have the latest and greatest drivers and firmware installed on them. It only happens during the working day, when those clients would be in use. My next port of call will therefore probably be to check for driver updates on those clients I guess. Bit of a sticky issue this....

[bulldog1]

Dhoward,

Well that's not good. I ran across an article talking about % Privileged Time. I started up Perf Mon on the system and added in % Privileged Time from the Object "Process". It's showing exactly what the processor is showing with the avg being 70-80%. I then found this hotfix from Microsoft An application stops responding, experiences low performance, or experiences high privileged CPU usage if many large I&#47O operations are performed in Windows 7 or Windows Server 2008 R2 . If have SP1 installed then this hotfix is included and it didn't fix the issue. We are not loading SP1 yet. I'm going to apply this tonight and see if it fixes the issue. Applied it on a non production system and it required a reboot. I'll let you know what happens.

[dhoward_westexetc]

Dhoward,

Well that's not good. I ran across an article talking about % Privileged Time. I started up Perf Mon on the system and added in % Privileged Time from the Object "Process". It's showing exactly what the processor is showing with the avg being 70-80%. I then found this hotfix from Microsoft An application stops responding, experiences low performance, or experiences high privileged CPU usage if many large I&#47O operations are performed in Windows 7 or Windows Server 2008 R2 . If have SP1 installed then this hotfix is included and it didn't fix the issue. We are not loading SP1 yet. I'm going to apply this tonight and see if it fixes the issue. Applied it on a non production system and it required a reboot. I'll let you know what happens.

 

If this does turn out to be the case bulldog1 then it would potentially only fix one of our servers. Our File server runs Storage Server 2008 SP2 and displays the srv 2012 errors, albeit on nowhere near the same frequency as our 2008 R2 one. The significant thing is both are used in a read context (the 2008 R2 one provides the redirected desktop and start menu for students to use).

[bulldog1]

Dhoward,

I've been monitoring the system today and the "System" Process (NT Kernel & System) has been averaging between 3% and 15%. This is down from the 60% - 80% that it was consuming before. It's still a little weird that this is the only system we have with this issue, but for now it looks like the issue is fixed or at least much better. So I would say the hotfix did the trick. Let me know what you see if you end up applying the hotfix. It looks like they have 3 version of the fix for Windows 7/Windows 2008 R2 (X86, x64, ia64) I applied the x64. I didn't know 2008 R2 had a 32bit x86 version, I thought it was only 2008 that had that. So not sure what that is all about.

[dhoward_westexetc]

Dhoward,

I've been monitoring the system today and the "System" Process (NT Kernel & System) has been averaging between 3% and 15%. This is down from the 60% - 80% that it was consuming before. It's still a little weird that this is the only system we have with this issue, but for now it looks like the issue is fixed or at least much better. So I would say the hotfix did the trick. Let me know what you see if you end up applying the hotfix. It looks like they have 3 version of the fix for Windows 7/Windows 2008 R2 (X86, x64, ia64) I applied the x64. I didn't know 2008 R2 had a 32bit x86 version, I thought it was only 2008 that had that. So not sure what that is all about.

 

The x86 patch will be for Windows 7 only, since that has a 32-bit version and they share the same issue clearly.

 

I'll bear your feedback in mind and schedule in applying the hotfix, but I don't know what we do if that same issue is causing the 2012s on our File Server. Thanks bulldog1.

[hboogz]

Hey dhoward -

 

Have you been able to pinpoint the issue? I'm having the same exact issue and tried everything under the sun on the serverOS side but believe it could be related to the older Symantec clients on the station.

[dhoward_westexetc]

Hey dhoward -

 

Have you been able to pinpoint the issue? I'm having the same exact issue and tried everything under the sun on the serverOS side but believe it could be related to the older Symantec clients on the station.

 

Sorry no news yet on this, will have to wait until September to see what has happened.

 

I have torn out the XP & McAfee 8.7i clients and replaced them with Windows 7 & McAfee 8.8, together with the unchecking of 'block connection when a threat is detected in a shared folder'. We have added a load of users to our 2008 file server in recent weeks, and srv errors haven't increased (in fact the number of srv errors has been slightly lower of recent). This indicates that it is not load-related, but network errors generated by clients (as the issue only happens during the school day, when certain groups of machines are in use). My hope is that the McAfee upgrade will have at least reduced the errors, if not eliminated them altogether. All of our 2008/R2 servers are running McAfee 8.8 now, and have up to date BIOS/RAID/NIC drivers.

 

We also had a number of troublesome ethernet cards in some of our systems that had dodgy XP drivers, we upgraded those drivers and things seem improved, but I think they may have been contributing to it. Those systems are now on Windows 7 with drivers 1 year newer than the drivers installed on the XP build. Those systems were generating several errors, to the extent that the managed switch was shutting the ports down due to the number of errors (requiring me to re-enable the ports)!

 

My port of call has been to keep the AV clients on the latest version, and look for any odd network behaviours on any clients.

[dhoward_westexetc]

I appear to have cured my srv 2012 errors.

 

It was a combination of installing McAfee 8.8 with the checkbox in on access scanner settings 'block the connection when a threat is detected in a shared folder'. With that setting enabled I could semi-reliably create the srv 2012 errors myself. We also had some PCs that had dodgy NIC drivers, but they were re-installed with Windows 7 SP1 and more up to date NIC drivers.

 

All our DCs, DNS and DHCP are 2008 R2 now.

 

So my advice to anyone having srvs:

 

• Update your NIC drivers
  
• Update server NIC and RAID drivers and firmware
  
• Update your Antivirus software on both client and server
  
• Check for any client PCs with any strange networking issues.
  

 

I also had to disable Network Access Protection on several of my DHCP scopes as our wireless system doesn't support NAP.

 

I think the fact that all of our clients are now Windows 7 SP1, and our 2008 R2 servers are now at SP1 aswell, is helping matters.

 

I hope this helps many of you cure your srv 2012 errors.