Event ID 2012 Source Srv on Server 2008

[cookie_monster]

I've been getting these errors on a few of my 2008 SP2 servers and i've not been able to get to the bottom of it. It says "Occassional errors are expected, but large amounts of these indicate a possible error in your network configuration.". It doesn't really give me an idea of how many errors might indicate a problem with my setup, I see about two-four per hour.

 

http://www.eventid.net/display.asp?eventid=2012&source=Srv

 

There's not much info out there about the error but a few posts have pointed the finger at AV software, I'm wondering if anyone else sees this error on their 2008 servers.

 

I'm pretty sure it's not a hardware or driver issues as i'm getting it on a physical HP server and on a couple of Xenserver VM's that are hosted on a Sun box. I haven't been able to rule out the switch yet but none of the 2003 servers on the same switch show the error. The only shared config other than the switch is the AV software NOD32. When I get chance i'll try a box without that.

 

Any other ideas?

 

Thanks.

[Richie1972]

I had this occurring a lot one one file server (2008 Enterprise X64) that was a Hyper-V guest running Sophos as its AV.

I upped the RAM and CPUs which reduced the frequency.

In the end, due to other problems, I ended up re-installing the server, so never really solved it

[PEO]

Hi, did you ever get to the bottom of this, Im having similar issues. i'm starting to think that its this dam stupid kaspersky antivirus

[dhoward_westexetc]

We are having this effect as well on 1 2008 R2 server (PowerEdge R710) and one Storage Server 2008 NAS appliance (PowerVault NX3100). Have tried everything I can think of, updated BIOS, re-installed NIC drivers, check all the settings etc, and I am totally lost. We get around the same frequency, several per hour that tend to happen in groups.

RAID drivers/firmware, HDD firmware is all up to date.

 

Our AV is McAfee VirusScan 8.7i Enterprise with patch 4 installed.

 

Anyone have any ideas on what else I should troubleshoot?

Edited October 21, 2010 by dhoward_westexetc
Added AV

[PiqueABoo]

Hmm.. there is definitely something in this one i.e. I've noticed it happening on both 32-bit 2008 and 64-bit 2008 R2 servers, but not very often on 2003. This is different sites, but common features are:

 

- They are all DCs.

- They all have McAfee 9.7i Patch 3 or 4, but if that's the cause then it's a serious regresssion because I remember that messing with the net redirector and causing this a few years ago - they eventually fixed it.

- Most have IPV6, SNP, new autotuning features and so on turned off.

 

I'm not seriously worried but I would like them gone. Something I haven't gotten around to yet is actually looking at the event data ... it might not be that useful but there is an error code i.e. extra clue, somewhere in there.

[dhoward_westexetc]

Hmm.. there is definitely something in this one i.e. I've noticed it happening on both 32-bit 2008 and 64-bit 2008 R2 servers, but not very often on 2003. This is different sites, but common features are:

 

- They are all DCs.

- They all have McAfee 9.7i Patch 3 or 4, but if that's the cause then it's a serious regresssion because I remember that messing with the net redirector and causing this a few years ago - they eventually fixed it.

- Most have IPV6, SNP, new autotuning features and so on turned off.

 

I'm not seriously worried but I would like them gone. Something I haven't gotten around to yet is actually looking at the event data ... it might not be that useful but there is an error code i.e. extra clue, somewhere in there.

 

Our problem servers aren't DCs, they are member servers. Our 2008 R2 DC does not exhibit this issue.

I haven't found the event data all that useful TBH. On my google searches all the event logs show similar byte data, but no real pinpoint to the cause.

 

Interesting you mention McAfee 8.7i - I heard several reports of AV causing this. But then I don't really want to disable AV on servers. Suppose it could be a useful test for a very short time. Trouble is 8.7i is the only McAfee AV that will work on Server 2008 R2.

 

What I have noticed is that the problem servers are ones with a lot of file access on network shares (one is our File server, the other one holds our menus for our redirected desktop/start menu for students), generally with heavy loads the problem appears, but outside of peak times the problem all but vanishes.

 

I'll probably look at the AV route on Friday....

Edited October 21, 2010 by dhoward_westexetc
added file access

[SYNACK]

TCP/IP offloading to the NIC going wrong maybe, especially if its passing through to a VM.

[PiqueABoo]

Good thought, but which TCP/IP offloading? Looking at a system that produces most of these events (most workstations & the busiest) I've just gone to the Intel 82574L Gigabit NIC advanced config and turned off:

 

- IPv4 Checksum Offload

- Large Send Offload (IPv4)

- TCP Checksum Offload (IPv4)

 

IIRC the TCP checksum first has crossed my path in the past on some 2003 boxes, the large send was part of someone else's fix for a VM scenario but I'm not fussy, and I added the IPv4 Checksum because I suppose I may as well knock out IP checksums as well as the TCP ones.

 

PS: Did this remotely and of course the connection dropped for each change - it's funny how much panicking you can fit into in a couple of seconds.

Edited October 21, 2010 by PiqueABoo
Added another disabled setting

[dhoward_westexetc]

Did those TCP offload changes on our 2008 R2 server and that brought it to its knees! Mind you it could have already gone wrong....

I'll do those changes again over half term and test them to ensure good connectivity.

 

I also notice there are 3 other hotfixes relating to Hyper-V, which is on our 2008 R2 box, KBs 981791, 974909 and 981836. I'll probably also try those and see if they make it any better. I think I will do those over half term though, given this morning's experience! We had the 981791 error last night.

 

I'll also test McAfee over half term as well.

[PiqueABoo]

Back to the drawing board (those offload changes didn't change anything for me). Can only think that something I'm doing must be causing these - I've got a new 2K8R2 box without AV on it yet, so if these 2012s haven't already started I'll watch out for them when AV goes on towards the end of next week.

[dhoward_westexetc]

Back to the drawing board (those offload changes didn't change anything for me). Can only think that something I'm doing must be causing these - I've got a new 2K8R2 box without AV on it yet, so if these 2012s haven't already started I'll watch out for them when AV goes on towards the end of next week.

 

That's useful to know, thanks for that. I'll ditch the idea at our end regarding the TCP/IP offload, and try instead ripping out McAfee on our menus server.

 

Let me know how it goes for you next week, particularly if you get the 2012s during half term. It sounds like you are in the same boat as me!

 

One more thing, does one of your servers that have the 2012s have Hyper-V installed on it? There is an update rollup for Hyper-V, dated 24 August 2010.

Edited October 22, 2010 by dhoward_westexetc
Hyper-V

[PiqueABoo]

No hyper-v involved, but that large send offload coupled with another change I forget was someone's solution somewhere for this event on a hyper-v guest.

 

+ I've looked at a few and so far for me they're all the Invalid Device State error code 0xc0000184

[dhoward_westexetc]

No hyper-v involved, but that large send offload coupled with another change I forget was someone's solution somewhere for this event on a hyper-v guest.

 

+ I've looked at a few and so far for me they're all the Invalid Device State error code 0xc0000184

 

I think all of ours are that one as well, i'll check on Monday. Chances are then if that is the case then there is the same cause. It will be interesting to see if you get the error when you install your AV.

[PEO]

In my case I have SAN involved. I managed to reduce mine by changing the network cards. At Peak times i.e a wednesday the schools network is heavily used and thats when I start to see the error more, pluss the system can slow to an almost hault.

 

I isolated all the network devices and fund duplicate Ip addresses on two switches and a switch with the sam IP as the UPS network card!!

 

will see if that makes any improvement and report back.

[jsnetman]

I was also getting these errors when first setting up our 2 DC's win server 2008r2. Never got to the bottom of the errors so decided to team all four nics on both servers and then load balance the connections across our switches in the main server room (never got the errors again). We also had one big subnet, had 600 PC's/Laptops and maybe 200 network devices on the same subnet, basically far too many for one subnet so created two extra vlans with routing to segment the broadcast traffic.

 

Problem being when we teamed the nics Hyper-V did not like the teamed nics with bromcom nic drivers. Think this has been fixed now so am going to unteam them install hyper-v then team them back up again to get hyper-v running again.

 

What I'm really indicating here is possible network design problems which server 08 is more sensitive to. I came to my current school when we had 150 pc's and a few printers so just had one small subnet for all devices. As the network grew to its present size I did not look at the design of the netwok and it gradually degraded almost to a stop.

Edited October 30, 2010 by jsnetman

[dhoward_westexetc]

I was also getting these errors when first setting up our 2 DC's win server 2008r2. Never got to the bottom of the errors so decided to team all four nics on both servers and then load balance the connections across our switches in the main server room (never got the errors again). We also had one big subnet, had 600 PC's/Laptops and maybe 200 network devices on the same subnet, basically far too many for one subnet so created two extra vlans with routing to segment the broadcast traffic.

 

Problem being when we teamed the nics Hyper-V did not like the teamed nics with bromcom nic drivers. Think this has been fixed now so am going to unteam them install hyper-v then team them back up again to get hyper-v running again.

 

What I'm really indicating here is possible network design problems which server 08 is more sensitive to. I came to my current school when we had 150 pc's and a few printers so just had one small subnet for all devices. As the network grew to its present size I did not look at the design of the netwok and it gradually degraded almost to a stop.

 

How do you team network adapters together in Server 2008 R2? I've heard that mentioned several times but never understood how that is actually done.

 

Our servers are on a separate VLAN from the other devices.

[jsnetman]

How do you team network adapters together in Server 2008 R2? I've heard that mentioned several times but never understood how that is actually done.

 

In our case Dell servers and broadcom NICS you download the broadcom utility to team them. Other network adaptors should have similar utilities. It is relatively easy to do. You will need to do it when the servers are'nt going to be online for 20/30 minutes. Would suggest a half term/weekend or after school.

Edited November 1, 2010 by jsnetman

[dhoward_westexetc]

In our case Dell servers and broadcom NICS you download the broadcom utility to team them. Other network adaptors should have similar utilities. It is relatively easy to do. You will need to do it when the servers are'nt going to be online for 20/30 minutes. Would suggest a half term/weekend or after school.

 

Presumably this will have to be in load balancing mode then? Does this still bind both network adapters to a single IP address?

[jsnetman]

Presumably this will have to be in load balancing mode then? Does this still bind both network adapters to a single IP address

 

Yes under our utility I chose smart load balancing and you team the nics then set one Ip on the teamed nics icon that shows up after teaming. In our case we did all four. You can do three and have one aas a fallover in case the team breaks, but this has never happened here. I suppose having one as a fallover means you won't lose any connection if you are tinkering with teams remotely.

Edited November 1, 2010 by jsnetman

[dhoward_westexetc]

Yes under our utility I chose smart load balancing and you team the nics then set one Ip on the teamed nics icon that shows up after teaming. In our case we did all four. You can do three and have one aas a fallover in case the team breaks, but this has never happened here. I suppose having one as a fallover means you won't lose any connection if you are tinkering with teams remotely.

 

I'm only thinking of having a load balance of two in our case, due to the fact that we do not have that many ports on our Gigabit switch. Plus our PowerVault server, which also has these errors, only has two NICs anyway, only our PowerEdge R710s have four NICs. I guess once we retire our older servers and the network cables come back we could do a team of three.

 

I'm just trying to find some redundant switch ports on our 100Mbps blades that I can move some devices into that don't need to be on our Gigabit switch, so that we have the maximum available on our Gigabit switch for our servers.

[dhoward_westexetc]

OK, I have now enabled teaming on the PowerVault server, where the problem is most concerning, using both NICs in smart load balance and failover mode.

 

There haven't been any unexpected consequences, and network connectivity was restored far faster than I expected. Performance after school is, as I expected, very good now.

 

Internal testing is good, but the acid test comes tomorrow! I will let you know how it goes.

[dhoward_westexetc]

Teaming doesn't seem to have done the trick, still getting them. I'll have to keep on monitoring the NIC event logs and gather more information.

 

Still, getting new information every time!

[jsnetman]

Have they not reduced, when I said never got them again you would probably still get the odd one, have you connected the nics to different switches i.e not on the same switch, we have several switches here which we connected each nic to a seperate switch on both servers. Just checked our event log and still get the odd ID 2012 but not a stream of them like before we teamed. Edited November 4, 2010 by jsnetman

[dhoward_westexetc]

Actually from monitoring they have reduced, quite substantially. we had two instances in 30 minutes, in that time there would normally be several.

 

It's tricky to be able to span across multiple switches in our case because we have a single passport chassis, and most of our switches are 100Mbps only, so if we were to put it onto another adapters they would both run at 100Mbps and not 1Gbps. So both are on our 1Gbps switch, its not ideal but it is the best we can do.

 

I have my colleague doing some load testing on it, copying 8GB of data to and from it during lesson time to try and put it under some load. I'll keep tabs on it.

[dhoward_westexetc]

OK, after half a day the number of srv errors has massively reduced over this time yesterday in the same period. We still get one or two every hour or so, but nothing like it was before, so I am gaining confidence. It is an improvement, which is what I have not seen for a long time.

 

However, what I am finding is that every now and again (probably 1 or two times a day), clients cannot connect into the PowerVault, but I can connect out from the PowerVault server. This will happen for about a minute, then suddenly all will be well again. This is accompanied by a series of about 4-5 consecutive srv errors. After that, there will be none for ages. Do you know what may be causing this?

 

I am also finding on the Broadcom control panel statistics is that the number of dropped receive packets, while still a tiny percentage of the total number of receive packets, is increasing constantly. The dropped packet percentage is around 0.2% for both adapters in the team, is that around normal?

 

From what I have read, AV does seem to play a role in this. Most of our client machines have McAfee 8.7i, but some have 8.5i. I'll be replacing most of the 8.5i installs with 8.7i, so I will see if that makes any difference to the number of errors. The new servers have 8.7i.

Edited November 4, 2010 by dhoward_westexetc
added dropped packets