+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 16
Network and Classroom Management Thread, Broadcast Storm in Technical; After running round like a headless chicken this morning for an hour because everything on the network ground to a ...
  1. #1
    HaleStorm's Avatar
    Join Date
    Jun 2008
    Location
    Sheffield
    Posts
    1,032
    Thank Post
    89
    Thanked 164 Times in 144 Posts
    Rep Power
    93

    Broadcast Storm

    After running round like a headless chicken this morning for an hour because everything on the network ground to a halt, we restarted EVERYTHING on networky (switches, servers, printers and Wireless points) and we believe something is causing a broadcast storm intermittently, is there any easy way of finding it? I've tried wireshark but it wont work when the storm is happening.
    And as we dont use managed switches at the moment its even harder to track down, so the network manager tells me anyway

    any ideas or common equipment that could cause such a pain in the bum?
    Cheers Guys

  2. #2

    SYNACK's Avatar
    Join Date
    Oct 2007
    Posts
    11,172
    Thank Post
    868
    Thanked 2,699 Times in 2,288 Posts
    Blog Entries
    11
    Rep Power
    772
    Could eb a network loop, check for looped back network cables. Double ports in classrooms are a favorite for people plugging in both ends of a patch lead and demanding that the network does not work.

    Best way to isolate it is to wait till its messing the network and then yank the connections between the switches one by one, checking after each to see if it has stopped. This should let you isolate it down to the switch that is is on and give you a lot less to check.

    This is one bit where managed switches really prove their worth as they have features built in that can mitigate this kind of fault. If your ones have even rudimentry web interfaces they may even have STP that you could switch on to help limit this kind of fault.

  3. Thanks to SYNACK from:

    HaleStorm (12th October 2011)

  4. #3
    HaleStorm's Avatar
    Join Date
    Jun 2008
    Location
    Sheffield
    Posts
    1,032
    Thank Post
    89
    Thanked 164 Times in 144 Posts
    Rep Power
    93
    We've just got to keep what we got running for 2more months and then we go CIVICA which brings a fully managed network hurrah

  5. #4

    TechMonkey's Avatar
    Join Date
    Dec 2005
    Location
    South East
    Posts
    3,287
    Thank Post
    225
    Thanked 405 Times in 302 Posts
    Rep Power
    162
    We ahve had a couple of these and luckily our set up keeps them contained to a segment of network. Not ideal but better than the whole network going down. The only way we have found to track it is to turn off all the switches, disconnect all connections and then slowly bring a switch up and plug bits in. Once it goes crazy or if you run a continuous ping on a nearby machine and it starts failing you know which area it is in. Then you either do the same for that areas cab or if that narrows it down to a room I would go to the room and unplug cables from the wall and plug them in one by one and test.

    Each time it has happened here it has been a dodgy or dying network card on a station.

  6. #5
    HaleStorm's Avatar
    Join Date
    Jun 2008
    Location
    Sheffield
    Posts
    1,032
    Thank Post
    89
    Thanked 164 Times in 144 Posts
    Rep Power
    93
    I have found the cause, hurrah! a combination of Print Manager Plus trying to constantly push "print rules" at printers and 2 dodgy Panasonic Network Cameras, turned them off and everything settled down pretty much instantly

  7. Thanks to HaleStorm from:

    AngryTechnician (12th October 2011)

  8. #6

    Join Date
    Apr 2011
    Location
    Manchester
    Posts
    173
    Thank Post
    10
    Thanked 8 Times in 7 Posts
    Rep Power
    21
    Hello,


    Hope you dont mind me hijacking this thread since you solved your problem.

    our ISP noticed out network switches broadcasting too often and sugested It get looked into because it was seen as unusually to him. his diagnosis was purely noticing the switching status LED's blinking in sync with each other.

    I know I've got alot to learn regarding networking but thats what I'm in the industry for. Until now I left if because I didnt know where to start.

    I downloaded wire shark 1.6.2 and straight away noticed the following

    Code:
    No.     Time            Source                Destination           Protocol Length Info
          1 12:59:44.742768 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.100?  Tell 192.168.1.2
          2 12:59:44.742770 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.16?  Tell 192.168.1.2
          3 12:59:44.742770 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.22?  Tell 192.168.1.2
          4 12:59:44.742811 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.235?  Tell 192.168.1.2
          5 12:59:44.742812 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.236?  Tell 192.168.1.2
          6 12:59:44.742812 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.29?  Tell 192.168.1.2
          7 12:59:44.742813 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.31?  Tell 192.168.1.2
          8 12:59:44.742813 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.33?  Tell 192.168.1.2
          9 12:59:44.742813 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.38?  Tell 192.168.1.2
         10 12:59:44.742813 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.44?  Tell 192.168.1.2
         11 12:59:44.742813 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.70?  Tell 192.168.1.2
         12 12:59:45.412563 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.100?  Tell 192.168.1.2
         13 12:59:45.412564 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.16?  Tell 192.168.1.2
         14 12:59:45.412565 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.22?  Tell 192.168.1.2
         15 12:59:45.413315 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.235?  Tell 192.168.1.2
         16 12:59:45.413316 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.236?  Tell 192.168.1.2
         17 12:59:45.413317 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.29?  Tell 192.168.1.2
         18 12:59:45.413317 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.31?  Tell 192.168.1.2
         19 12:59:45.413318 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.33?  Tell 192.168.1.2
         20 12:59:45.413318 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.38?  Tell 192.168.1.2
         21 12:59:45.413319 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.44?  Tell 192.168.1.2
         22 12:59:45.413319 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.70?  Tell 192.168.1.2
         24 12:59:45.611839 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.95?  Tell 192.168.1.3
         59 12:59:46.412531 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.100?  Tell 192.168.1.2
         60 12:59:46.412532 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.16?  Tell 192.168.1.2
         61 12:59:46.412533 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.22?  Tell 192.168.1.2
         62 12:59:46.413271 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.235?  Tell 192.168.1.2
         63 12:59:46.413271 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.236?  Tell 192.168.1.2
         64 12:59:46.413272 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.29?  Tell 192.168.1.2
         65 12:59:46.413272 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.31?  Tell 192.168.1.2
         66 12:59:46.413272 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.33?  Tell 192.168.1.2
         67 12:59:46.413273 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.38?  Tell 192.168.1.2
         68 12:59:46.413273 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.44?  Tell 192.168.1.2
         69 12:59:46.413273 IntelCor_0b:b3:a4     Broadcast             ARP      60     Who has 192.168.1.70?  Tell 192.168.1.2
         80 12:59:48.366589 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.95?  Tell 192.168.1.3
         81 12:59:49.110848 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.95?  Tell 192.168.1.3
         83 12:59:50.110864 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.95?  Tell 192.168.1.3
        146 12:59:52.110584 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.43?  Tell 192.168.1.3
        177 12:59:54.866488 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.43?  Tell 192.168.1.3
        179 12:59:55.612297 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.43?  Tell 192.168.1.3
        181 12:59:56.610828 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.43?  Tell 192.168.1.3
        258 12:59:58.369865 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        298 12:59:59.110496 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        300 13:00:00.110756 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        302 13:00:01.366468 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        304 13:00:02.110791 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        305 13:00:02.213143 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.30?  Tell 192.168.1.3
        312 13:00:03.111644 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.143?  Tell 192.168.1.3
        388 13:00:04.872719 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.41?  Tell 192.168.1.3
        389 13:00:04.872720 FujitsuT_67:10:6c     Broadcast             ARP      60     Who has 192.168.1.89?  Tell 192.168.1.3
    the list is alot longer and I can provide more infomation on request.

    my only guess is this isn't normal behaviour.
    192.168.1.2 and 192.168.1.3 are both of our servers. it's half term so all other workstations are off except from a few teachers left on from friday and I've not noticed them yet but I've just RDP's to them to shut them down.

    I ran wireshark from my client machine in the staffroom, What steps can I take to work out what is causing the excessive broadcasts?

    Thanks

    Bryan

  9. #7
    ChrisH's Avatar
    Join Date
    Jun 2005
    Location
    East Lancs
    Posts
    5,009
    Thank Post
    120
    Thanked 282 Times in 260 Posts
    Rep Power
    108
    Is 192.168.1.3 a server or a client running some kind of classroom management software?

  10. #8

    Join Date
    Apr 2011
    Location
    Manchester
    Posts
    173
    Thank Post
    10
    Thanked 8 Times in 7 Posts
    Rep Power
    21
    ..1.3 is our 2008 64bit server which run AD, DNS, and network shares, I do have spiceworks running on it but figured that since ..1.2 is broadcasting equally as much it wouldn't be that.

  11. #9

    m25man's Avatar
    Join Date
    Oct 2005
    Location
    Romford, Essex
    Posts
    1,626
    Thank Post
    49
    Thanked 460 Times in 336 Posts
    Rep Power
    140
    Your servers 192.168.1.2 and 192.168.1.3 will only send a broadcast or ARP request when DNS or Host lookup fails.

    If you have excessive broadcasts being made from your servers I would look examine your DHCP/DNS configs carefully as these should be negating the need to broadcast.

    Most of the time it comes back to a simple typo or missed box in the configuration that causes this. Stale reverse DNS data and scavenging issues giving multiple host references especially.

    Spiceworks has a discovery process that uses ARP to determine if you have DNS issues and reports as such if there are problems so it could well be linked but should not be enough to be considered a storm.

  12. #10


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by m25man View Post
    Your servers 192.168.1.2 and 192.168.1.3 will only send a broadcast or ARP request when DNS or Host lookup fails.
    ?? Are you sure about this?
    ARP maps layer 2 to layer 3.
    How does anything on the network know the mac to IP translation of anything else without doing an ARP broadcast.

    I'd say it was perfectly normal behavior, esp as it is the servers performing the broadcast.

    @RageRiot - whats teh timescale of your wireshark snapshot?

  13. #11

    Join Date
    Apr 2011
    Location
    Manchester
    Posts
    173
    Thank Post
    10
    Thanked 8 Times in 7 Posts
    Rep Power
    21
    its constant. it goes through a loop it seems. i set wireshark to filter arp and its a constant stream.

  14. #12

    Join Date
    Apr 2011
    Location
    Manchester
    Posts
    173
    Thank Post
    10
    Thanked 8 Times in 7 Posts
    Rep Power
    21
    i can post alot more lines from the capture i just selected a bunch of them to get the point across.

  15. #13


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by RageRiot View Post
    its constant. it goes through a loop it seems. i set wireshark to filter arp and its a constant stream.



    Edit: sorry I only just looked that the wireshark timestamp is on your post.

    It does look like excessive traffic - How big is the netwok? is it segregated with VLAN?
    Last edited by CyberNerd; 24th October 2011 at 05:46 PM.

  16. #14


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    8 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2
    19 12:59:45.413318 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2
    66 12:59:46.413272 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2
    This doesn't look good at all, all to the same address in a fraction of a second - have you correctly enabled STP on your switches?
    I'm not sure if this could be a sign of an ARP flood attack?

  17. #15

    m25man's Avatar
    Join Date
    Oct 2005
    Location
    Romford, Essex
    Posts
    1,626
    Thank Post
    49
    Thanked 460 Times in 336 Posts
    Rep Power
    140
    Quote Originally Posted by CyberNerd View Post
    ?? Are you sure about this?
    ARP maps layer 2 to layer 3.
    How does anything on the network know the mac to IP translation of anything else without doing an ARP broadcast.

    I'd say it was perfectly normal behavior, esp as it is the servers performing the broadcast.

    @RageRiot - whats teh timescale of your wireshark snapshot?
    Maybe I didnt explain that well enough,
    @CyberNerd is quite correct, but the servers build and maintain their own ARP cache.
    The ARP cache contains IP address / MAC address translations so that every time an IP packet are to be sent, the MAC address doesn't have to queried through a broadcast, instead it can use the cached address.

    Windows name resolution is always DNS > Hosts File > Broadcast

    If the first two fail or are unavailable the default is to Broadcast, therefore when DNS is in trouble the first thing to go through the roof is broadcast traffic.

    Clients on the network request resolution of network objects by making DNS requests to the server, the server would normally return valid DNS data but if the information is unavailable or incorrect the server will in turn make the appropriate ARP in the hope for a response.
    This is usually accompanied by hundreds of clients doing the same thing as they start trying to resolve MAC to IP and IP to Netbios.

    ARP requests are "Normal" on any server just to maintain the MAC to IP tables but when they reach this level its normally a sign there are DNS issues elsewhere.

    It need not even be the server itself, I have seen devices with bad network information, wrong DNS, Gateway or subnet mask settings all result in the inability to resolve a host correctly which in turn propagates the broadcast traffic.

    Fortunately for me its a 2 minute job as I just plug in the Etherscope and a few minutes later it will tell me which device has the wrong settings....

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Broadcast Storm LLMNR
    By BooBoo in forum Windows Server 2008
    Replies: 2
    Last Post: 23rd June 2009, 01:31 PM
  2. HP 2650 broadcast storms (stp?)
    By jrubinstein in forum Wireless Networks
    Replies: 2
    Last Post: 19th August 2007, 08:51 AM
  3. Live broadcasting in primary school? How & Help!!
    By kfq61 in forum Educational Software
    Replies: 4
    Last Post: 10th July 2006, 03:35 PM
  4. live broadcast of poptech 2005 starts today
    By russdev in forum General Chat
    Replies: 3
    Last Post: 19th October 2005, 01:29 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •