Optiplex 9020 systems spewing IPv6 multicast traffic while asleep causing havok

[TG_MI]

I recently ran across a bizarre problem and I thought others might benefit from what we seem to have found.

 

SYMPTOMS:

Our initial "symptom" of the issue was an intermittent problem with various networked HP Laserjet printers (specifically LJ P2055DN, LJ P4015DN) whereby they would lock up, stop processing jobs, not even respond to button presses on the front panel. Remove the network cable and BOOM, the printer would begin responding to button presses, menu navigation, could even print out a configuration page. Plug the network cable back in and BOOM, instantly locked up again. Three different printers, same issue. The lockups would eventually subside on their own, only to happen again at a later date.

 

TROUBLESHOOTING:

Tried all of the typical stuff with the printers, thinking they were the root cause. Firmware updates, cold resets, different drivers etc. The problem came and went mysteriously. Once we realized that three separate networked printers on the same VLAN were exhibiting the same strange behavior (albeit sporadically) we began to focus on the network.

 

I began working with one of our network admins who asked me to call him the next time we received a complaint with one of the printers, which I did. He was able to observe that two different systems on the VLAN were spewing an excessive amount of IPv6 multicast listener discovery packets during the printer lockup; I believe he said a rate above 16,000 packets/s originating from two separate boxes on our network.

 

Armed with the IPs of the trouble making boxes, I set out to track them down. I expected to find a compromised system or someone using P2P software, maybe a box with Bonjour going nuts with multicast traffic…something. What I found was that the two systems were both relatively new Dell Optiplex 9020’s installed in classroom podiums. That immediately confused me because our classroom systems are locked down (no end users are administrators, managed patches etc) and the deployment applications are plain vanilla! Ran some scans, used some Sysinternals tools to look for anything malicious and/or chatty that would explain the multicast traffic and came up with ziltch. These systems were clean.

 

The network admin had provided me with charts showing when the traffic spikes took place on both boxes, so I decided to check the Windows eventlogs to see if I could find a correlation between the network activity and a user or application. I was surprised to find with 100% certainty that as far as Windows was concerned, both systems had entered a sleep state for the entire duration of the IPv6 multicast listener discovery packets. It wasn’t even close really, both systems had gone to sleep and remained asleep during the whole thing.

 

A quick google search of "optiplex 9020 sleep multicast" turned up several posts from around the web reporting the exact same behavior.

Opti 9020s with Intel I217-LM nics causing network issues - Desktop General Hardware Forum - Desktop - Dell Community

 

Dell Optiplex 9020 blasting "ICMPv6 Multicast Listener Discovery during S1 sleep - Spiceworks

 

The Dell Optiplex 9020 DDOS ICMPv6 Multicast Listener Discovery - [H]ard|Forum

 

CONCLUSION:

From what I gathered from reading, Dell Optiplex 9020 systems with the Intel I217-LM NIC with a BIOS revision prior to A05 were exhibiting this strange behavior – randomly flooding the VLAN with IPv6 multicast traffic for a period of time while the systems were in a sleep state. This past summer, we had purchased Optiplex 9020’s to replace all of our classroom podium systems (about 30 in the building that had the printer issues, all on the same VLAN).

 

 

I can’t yet confirm that upgrading to A05 will correct the problem (just figured this out with the network admin this morning) but others have seen some success. They do mention thought that WOL doesn’t work in A05.

 

SUMMARY:

If you’re experiencing unexplained lockups or sluggish performance from networked devices like printers, access points, NAS’s etc, have a network admin check for unexplained IPv6 multicast traffic, especially if you have Dell Optiplex 9020’s (possibly other devices with an Intel I217-LM NIC) on your network with an early BIOS revision as they may be sporadically spewing IPv6 multicast traffic (while in a sleep state) causing adverse effects on other devices.

Edited February 20, 2014 by TG_MI

[seawolf]

I recently ran across a bizarre problem and I thought others might benefit from what we seem to have found.

 

SYMPTOMS:

Our initial "symptom" of the issue was an intermittent problem with various networked HP Laserjet printers (specifically LJ P2055DN, LJ P4015DN) whereby they would lock up, stop processing jobs, not even respond to button presses on the front panel. Remove the network cable and BOOM, the printer would begin responding to button presses, menu navigation, could even print out a configuration page. Plug the network cable back in and BOOM, instantly locked up again. Three different printers, same issue. The lockups would eventually subside on their own, only to happen again at a later date.

 

TROUBLESHOOTING:

Tried all of the typical stuff with the printers, thinking they were the root cause. Firmware updates, cold resets, different drivers etc. The problem came and went mysteriously. Once we realized that three separate networked printers on the same VLAN were exhibiting the same strange behavior (albeit sporadically) we began to focus on the network.

 

I began working with one of our network admins who asked me to call him the next time we received a complaint with one of the printers, which I did. He was able to observe that two different systems on the VLAN were spewing an excessive amount of IPv6 multicast listener discovery packets during the printer lockup; I believe he said a rate above 16,000 packets/s originating from two separate boxes on our network.

 

Armed with the IPs of the trouble making boxes, I set out to track them down. I expected to find a compromised system or someone using P2P software, maybe a box with Bonjour going nuts with multicast traffic…something. What I found was that the two systems were both relatively new Dell Optiplex 9020’s installed in classroom podiums. That immediately confused me because our classroom systems are locked down (no end users are administrators, managed patches etc) and the deployment applications are plain vanilla! Ran some scans, used some Sysinternals tools to look for anything malicious and/or chatty that would explain the multicast traffic and came up with ziltch. These systems were clean.

 

The network admin had provided me with charts showing when the traffic spikes took place on both boxes, so I decided to check the Windows eventlogs to see if I could find a correlation between the network activity and a user or application. I was surprised to find with 100% certainty that as far as Windows was concerned, both systems had entered a sleep state for the entire duration of the IPv6 multicast listener discovery packets. It wasn’t even close really, both systems had gone to sleep and remained asleep during the whole thing.

 

A quick google search of "optiplex 9020 sleep multicast" turned up several posts from around the web reporting the exact same behavior.

Opti 9020s with Intel I217-LM nics causing network issues - Desktop General Hardware Forum - Desktop - Dell Community

 

Dell Optiplex 9020 blasting "ICMPv6 Multicast Listener Discovery during S1 sleep - Spiceworks

 

The Dell Optiplex 9020 DDOS ICMPv6 Multicast Listener Discovery - [H]ard|Forum

 

CONCLUSION:

From what I gathered from reading, Dell Optiplex 9020 systems with the Intel I217-LM NIC with a BIOS revision prior to A05 were exhibiting this strange behavior – randomly flooding the VLAN with IPv6 multicast traffic for a period of time while the systems were in a sleep state. This past summer, we had purchased Optiplex 9020’s to replace all of our classroom podium systems (about 30 in the building that had the printer issues, all on the same VLAN).

 

 

I can’t yet confirm that upgrading to A05 will correct the problem (just figured this out with the network admin this morning) but others have seen some success. They do mention thought that WOL doesn’t work in A05.

 

SUMMARY:

If you’re experiencing unexplained lockups or sluggish performance from networked devices like printers, access points, NAS’s etc, have a network admin check for unexplained IPv6 multicast traffic, especially if you have Dell Optiplex 9020’s (possibly other devices with an Intel I217-LM NIC) on your network with an early BIOS revision as they may be sporadically spewing IPv6 multicast traffic causing adverse effects on other devices.

 

Interesting findings with Optiplex 9020s. Have you considered just disabling IPV6 on the adapter in Windows. We do this as part of our standard image, if you aren't using IPV6 there's no reason to have any IPV6 traffic on your network that aren't necessary.

[TG_MI]

I've read that suggestion and considered it yes, but if we can solve the issue with a BIOS update to the unit, I think that would be our preferred method for now. Long term, it just seems like a less thorny fix. I'll report back after we've investigated the resolution more thoroughly.

[FN-GM]

Do you use SCCM? If so you could use that to push out your BIOS update. I updated Dell 500 machines with an updated BIOS and it worked a treat.

[seawolf]

I've read that suggestion and considered it yes, but if we can solve the issue with a BIOS update to the unit, I think that would be our preferred method for now. Long term, it just seems like a less thorny fix. I'll report back after we've investigated the resolution more thoroughly.

 

Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?

[FN-GM]

Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?

 

The BIOS on the Optiplex range is designed to be updated on mass. Also disabling the IP v6 settings can cause issues with other applications.

[seawolf]

The BIOS on the Optiplex range is designed to be updated on mass. Also disabling the IP v6 settings can cause issues with other applications.

 

Choose your poison I suppose. I've never seen any issues here with any applications when IPV6 is disabled.

[TG_MI]

A bit more scouring of the net seems to suggest that this problem is seen in other pieces of hardware that have an Intel I217-LM NIC. So it's not just limited to the Dell Optiplex 9020's.

 

https://communities.intel.com/message/220048

ICMPv6 'Multicast Listener Report' messages are flooding the local network

[john]

Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?

 

That doesn't fully disable IPV6 you need to do a registry modification to do so

 

How to disable IPv6 or its components in Windows

 

Has some fixit's as well as the regkeys

[TG_MI]

It's not clear to me that disabling IPv6 at the OS level would prevent the NIC from misbehaving when the system is in a sleep state. This seems to be a firmware level issue, though I could be wrong.