+ Post New Thread
Results 1 to 10 of 10
Hardware Thread, Optiplex 9020 systems spewing IPv6 multicast traffic while asleep causing havok in Technical; I recently ran across a bizarre problem and I thought others might benefit from what we seem to have found. ...
  1. #1

    Join Date
    Jul 2013
    Posts
    16
    Thank Post
    0
    Thanked 4 Times in 4 Posts
    Rep Power
    3

    Optiplex 9020 systems spewing IPv6 multicast traffic while asleep causing havok

    I recently ran across a bizarre problem and I thought others might benefit from what we seem to have found.

    SYMPTOMS:
    Our initial "symptom" of the issue was an intermittent problem with various networked HP Laserjet printers (specifically LJ P2055DN, LJ P4015DN) whereby they would lock up, stop processing jobs, not even respond to button presses on the front panel. Remove the network cable and BOOM, the printer would begin responding to button presses, menu navigation, could even print out a configuration page. Plug the network cable back in and BOOM, instantly locked up again. Three different printers, same issue. The lockups would eventually subside on their own, only to happen again at a later date.

    TROUBLESHOOTING:
    Tried all of the typical stuff with the printers, thinking they were the root cause. Firmware updates, cold resets, different drivers etc. The problem came and went mysteriously. Once we realized that three separate networked printers on the same VLAN were exhibiting the same strange behavior (albeit sporadically) we began to focus on the network.

    I began working with one of our network admins who asked me to call him the next time we received a complaint with one of the printers, which I did. He was able to observe that two different systems on the VLAN were spewing an excessive amount of IPv6 multicast listener discovery packets during the printer lockup; I believe he said a rate above 16,000 packets/s originating from two separate boxes on our network.

    Armed with the IPs of the trouble making boxes, I set out to track them down. I expected to find a compromised system or someone using P2P software, maybe a box with Bonjour going nuts with multicast traffic…something. What I found was that the two systems were both relatively new Dell Optiplex 9020’s installed in classroom podiums. That immediately confused me because our classroom systems are locked down (no end users are administrators, managed patches etc) and the deployment applications are plain vanilla! Ran some scans, used some Sysinternals tools to look for anything malicious and/or chatty that would explain the multicast traffic and came up with ziltch. These systems were clean.

    The network admin had provided me with charts showing when the traffic spikes took place on both boxes, so I decided to check the Windows eventlogs to see if I could find a correlation between the network activity and a user or application. I was surprised to find with 100% certainty that as far as Windows was concerned, both systems had entered a sleep state for the entire duration of the IPv6 multicast listener discovery packets. It wasn’t even close really, both systems had gone to sleep and remained asleep during the whole thing.

    A quick google search of "optiplex 9020 sleep multicast" turned up several posts from around the web reporting the exact same behavior.
    Opti 9020s with Intel I217-LM nics causing network issues - Desktop General Hardware Forum - Desktop - Dell Community

    Dell Optiplex 9020 blasting "ICMPv6 Multicast Listener Discovery during S1 sleep - Spiceworks

    The Dell Optiplex 9020 DDOS ICMPv6 Multicast Listener Discovery - [H]ard|Forum

    CONCLUSION:
    From what I gathered from reading, Dell Optiplex 9020 systems with the Intel I217-LM NIC with a BIOS revision prior to A05 were exhibiting this strange behavior – randomly flooding the VLAN with IPv6 multicast traffic for a period of time while the systems were in a sleep state. This past summer, we had purchased Optiplex 9020’s to replace all of our classroom podium systems (about 30 in the building that had the printer issues, all on the same VLAN).


    I can’t yet confirm that upgrading to A05 will correct the problem (just figured this out with the network admin this morning) but others have seen some success. They do mention thought that WOL doesn’t work in A05.

    SUMMARY:
    If you’re experiencing unexplained lockups or sluggish performance from networked devices like printers, access points, NAS’s etc, have a network admin check for unexplained IPv6 multicast traffic, especially if you have Dell Optiplex 9020’s (possibly other devices with an Intel I217-LM NIC) on your network with an early BIOS revision as they may be sporadically spewing IPv6 multicast traffic (while in a sleep state) causing adverse effects on other devices.
    Last edited by TG_MI; 20th February 2014 at 08:20 PM.

  2. Thanks to TG_MI from:

    AngryTechnician (24th February 2014)

  3. #2

    seawolf's Avatar
    Join Date
    Jan 2010
    Location
    Melbourne
    Posts
    903
    Thank Post
    12
    Thanked 272 Times in 207 Posts
    Blog Entries
    1
    Rep Power
    173
    Quote Originally Posted by TG_MI View Post
    I recently ran across a bizarre problem and I thought others might benefit from what we seem to have found.

    SYMPTOMS:
    Our initial "symptom" of the issue was an intermittent problem with various networked HP Laserjet printers (specifically LJ P2055DN, LJ P4015DN) whereby they would lock up, stop processing jobs, not even respond to button presses on the front panel. Remove the network cable and BOOM, the printer would begin responding to button presses, menu navigation, could even print out a configuration page. Plug the network cable back in and BOOM, instantly locked up again. Three different printers, same issue. The lockups would eventually subside on their own, only to happen again at a later date.

    TROUBLESHOOTING:
    Tried all of the typical stuff with the printers, thinking they were the root cause. Firmware updates, cold resets, different drivers etc. The problem came and went mysteriously. Once we realized that three separate networked printers on the same VLAN were exhibiting the same strange behavior (albeit sporadically) we began to focus on the network.

    I began working with one of our network admins who asked me to call him the next time we received a complaint with one of the printers, which I did. He was able to observe that two different systems on the VLAN were spewing an excessive amount of IPv6 multicast listener discovery packets during the printer lockup; I believe he said a rate above 16,000 packets/s originating from two separate boxes on our network.

    Armed with the IPs of the trouble making boxes, I set out to track them down. I expected to find a compromised system or someone using P2P software, maybe a box with Bonjour going nuts with multicast traffic…something. What I found was that the two systems were both relatively new Dell Optiplex 9020’s installed in classroom podiums. That immediately confused me because our classroom systems are locked down (no end users are administrators, managed patches etc) and the deployment applications are plain vanilla! Ran some scans, used some Sysinternals tools to look for anything malicious and/or chatty that would explain the multicast traffic and came up with ziltch. These systems were clean.

    The network admin had provided me with charts showing when the traffic spikes took place on both boxes, so I decided to check the Windows eventlogs to see if I could find a correlation between the network activity and a user or application. I was surprised to find with 100% certainty that as far as Windows was concerned, both systems had entered a sleep state for the entire duration of the IPv6 multicast listener discovery packets. It wasn’t even close really, both systems had gone to sleep and remained asleep during the whole thing.

    A quick google search of "optiplex 9020 sleep multicast" turned up several posts from around the web reporting the exact same behavior.
    Opti 9020s with Intel I217-LM nics causing network issues - Desktop General Hardware Forum - Desktop - Dell Community

    Dell Optiplex 9020 blasting "ICMPv6 Multicast Listener Discovery during S1 sleep - Spiceworks

    The Dell Optiplex 9020 DDOS ICMPv6 Multicast Listener Discovery - [H]ard|Forum

    CONCLUSION:
    From what I gathered from reading, Dell Optiplex 9020 systems with the Intel I217-LM NIC with a BIOS revision prior to A05 were exhibiting this strange behavior – randomly flooding the VLAN with IPv6 multicast traffic for a period of time while the systems were in a sleep state. This past summer, we had purchased Optiplex 9020’s to replace all of our classroom podium systems (about 30 in the building that had the printer issues, all on the same VLAN).


    I can’t yet confirm that upgrading to A05 will correct the problem (just figured this out with the network admin this morning) but others have seen some success. They do mention thought that WOL doesn’t work in A05.

    SUMMARY:
    If you’re experiencing unexplained lockups or sluggish performance from networked devices like printers, access points, NAS’s etc, have a network admin check for unexplained IPv6 multicast traffic, especially if you have Dell Optiplex 9020’s (possibly other devices with an Intel I217-LM NIC) on your network with an early BIOS revision as they may be sporadically spewing IPv6 multicast traffic causing adverse effects on other devices.
    Interesting findings with Optiplex 9020s. Have you considered just disabling IPV6 on the adapter in Windows. We do this as part of our standard image, if you aren't using IPV6 there's no reason to have any IPV6 traffic on your network that aren't necessary.

  4. #3

    Join Date
    Jul 2013
    Posts
    16
    Thank Post
    0
    Thanked 4 Times in 4 Posts
    Rep Power
    3
    I've read that suggestion and considered it yes, but if we can solve the issue with a BIOS update to the unit, I think that would be our preferred method for now. Long term, it just seems like a less thorny fix. I'll report back after we've investigated the resolution more thoroughly.

  5. #4

    FN-GM's Avatar
    Join Date
    Jun 2007
    Location
    UK
    Posts
    15,761
    Thank Post
    825
    Thanked 1,663 Times in 1,448 Posts
    Blog Entries
    11
    Rep Power
    442
    Do you use SCCM? If so you could use that to push out your BIOS update. I updated Dell 500 machines with an updated BIOS and it worked a treat.

  6. #5

    seawolf's Avatar
    Join Date
    Jan 2010
    Location
    Melbourne
    Posts
    903
    Thank Post
    12
    Thanked 272 Times in 207 Posts
    Blog Entries
    1
    Rep Power
    173
    Quote Originally Posted by TG_MI View Post
    I've read that suggestion and considered it yes, but if we can solve the issue with a BIOS update to the unit, I think that would be our preferred method for now. Long term, it just seems like a less thorny fix. I'll report back after we've investigated the resolution more thoroughly.
    Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?

  7. #6

    FN-GM's Avatar
    Join Date
    Jun 2007
    Location
    UK
    Posts
    15,761
    Thank Post
    825
    Thanked 1,663 Times in 1,448 Posts
    Blog Entries
    11
    Rep Power
    442
    Quote Originally Posted by seawolf View Post
    Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?
    The BIOS on the Optiplex range is designed to be updated on mass. Also disabling the IP v6 settings can cause issues with other applications.

  8. #7

    seawolf's Avatar
    Join Date
    Jan 2010
    Location
    Melbourne
    Posts
    903
    Thank Post
    12
    Thanked 272 Times in 207 Posts
    Blog Entries
    1
    Rep Power
    173
    Quote Originally Posted by FN-GM View Post
    The BIOS on the Optiplex range is designed to be updated on mass. Also disabling the IP v6 settings can cause issues with other applications.
    Choose your poison I suppose. I've never seen any issues here with any applications when IPV6 is disabled.

  9. #8

    Join Date
    Jul 2013
    Posts
    16
    Thank Post
    0
    Thanked 4 Times in 4 Posts
    Rep Power
    3
    A bit more scouring of the net seems to suggest that this problem is seen in other pieces of hardware that have an Intel I217-LM NIC. So it's not just limited to the Dell Optiplex 9020's.

    https://communities.intel.com/message/220048
    ICMPv6 'Multicast Listener Report' messages are flooding the local network

  10. #9

    john's Avatar
    Join Date
    Sep 2005
    Location
    London
    Posts
    10,498
    Thank Post
    1,488
    Thanked 1,049 Times in 918 Posts
    Rep Power
    301
    Quote Originally Posted by seawolf View Post
    Disabling IPV6 is just a tick box in the adaptor properties. How would a BIOS update be less thorny. Seems much more risky and time consuming to me?
    That doesn't fully disable IPV6 you need to do a registry modification to do so

    How to disable IPv6 or its components in Windows

    Has some fixit's as well as the regkeys

  11. #10

    Join Date
    Jul 2013
    Posts
    16
    Thank Post
    0
    Thanked 4 Times in 4 Posts
    Rep Power
    3
    It's not clear to me that disabling IPv6 at the OS level would prevent the NIC from misbehaving when the system is in a sleep state. This seems to be a firmware level issue, though I could be wrong.

SHARE:
+ Post New Thread

Similar Threads

  1. [SCCM 2012] Optiplex 9020 Not Building?
    By DaveP in forum O/S Deployment
    Replies: 12
    Last Post: 6th November 2013, 09:59 AM
  2. Dell OptiPlex System Hard Drives
    By MatthewL in forum Hardware
    Replies: 6
    Last Post: 19th March 2009, 01:43 PM
  3. Anyone use the Bromcom system?
    By tarquel in forum MIS Systems
    Replies: 38
    Last Post: 25th May 2007, 04:11 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •