HaleStorm Posted October 12, 2011 Posted October 12, 2011 After running round like a headless chicken this morning for an hour because everything on the network ground to a halt, we restarted EVERYTHING on networky (switches, servers, printers and Wireless points) and we believe something is causing a broadcast storm intermittently, is there any easy way of finding it? I've tried wireshark but it wont work when the storm is happening. And as we dont use managed switches at the moment its even harder to track down, so the network manager tells me anyway any ideas or common equipment that could cause such a pain in the bum? Cheers Guys
SYNACK Posted October 12, 2011 Posted October 12, 2011 Could eb a network loop, check for looped back network cables. Double ports in classrooms are a favorite for people plugging in both ends of a patch lead and demanding that the network does not work. Best way to isolate it is to wait till its messing the network and then yank the connections between the switches one by one, checking after each to see if it has stopped. This should let you isolate it down to the switch that is is on and give you a lot less to check. This is one bit where managed switches really prove their worth as they have features built in that can mitigate this kind of fault. If your ones have even rudimentry web interfaces they may even have STP that you could switch on to help limit this kind of fault. 1
HaleStorm Posted October 12, 2011 Author Posted October 12, 2011 We've just got to keep what we got running for 2more months and then we go CIVICA which brings a fully managed network hurrah
TechMonkey Posted October 12, 2011 Posted October 12, 2011 We ahve had a couple of these and luckily our set up keeps them contained to a segment of network. Not ideal but better than the whole network going down. The only way we have found to track it is to turn off all the switches, disconnect all connections and then slowly bring a switch up and plug bits in. Once it goes crazy or if you run a continuous ping on a nearby machine and it starts failing you know which area it is in. Then you either do the same for that areas cab or if that narrows it down to a room I would go to the room and unplug cables from the wall and plug them in one by one and test. Each time it has happened here it has been a dodgy or dying network card on a station.
HaleStorm Posted October 12, 2011 Author Posted October 12, 2011 I have found the cause, hurrah! a combination of Print Manager Plus trying to constantly push "print rules" at printers and 2 dodgy Panasonic Network Cameras, turned them off and everything settled down pretty much instantly 1
RageRiot Posted October 24, 2011 Posted October 24, 2011 Hello, Hope you dont mind me hijacking this thread since you solved your problem. our ISP noticed out network switches broadcasting too often and sugested It get looked into because it was seen as unusually to him. his diagnosis was purely noticing the switching status LED's blinking in sync with each other. I know I've got alot to learn regarding networking but thats what I'm in the industry for. Until now I left if because I didnt know where to start. I downloaded wire shark 1.6.2 and straight away noticed the following No. Time Source Destination Protocol Length Info 1 12:59:44.742768 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.100? Tell 192.168.1.2 2 12:59:44.742770 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.16? Tell 192.168.1.2 3 12:59:44.742770 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.22? Tell 192.168.1.2 4 12:59:44.742811 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.235? Tell 192.168.1.2 5 12:59:44.742812 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.236? Tell 192.168.1.2 6 12:59:44.742812 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.29? Tell 192.168.1.2 7 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.31? Tell 192.168.1.2 8 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 9 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.38? Tell 192.168.1.2 10 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.44? Tell 192.168.1.2 11 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.70? Tell 192.168.1.2 12 12:59:45.412563 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.100? Tell 192.168.1.2 13 12:59:45.412564 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.16? Tell 192.168.1.2 14 12:59:45.412565 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.22? Tell 192.168.1.2 15 12:59:45.413315 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.235? Tell 192.168.1.2 16 12:59:45.413316 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.236? Tell 192.168.1.2 17 12:59:45.413317 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.29? Tell 192.168.1.2 18 12:59:45.413317 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.31? Tell 192.168.1.2 19 12:59:45.413318 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 20 12:59:45.413318 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.38? Tell 192.168.1.2 21 12:59:45.413319 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.44? Tell 192.168.1.2 22 12:59:45.413319 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.70? Tell 192.168.1.2 24 12:59:45.611839 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.95? Tell 192.168.1.3 59 12:59:46.412531 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.100? Tell 192.168.1.2 60 12:59:46.412532 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.16? Tell 192.168.1.2 61 12:59:46.412533 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.22? Tell 192.168.1.2 62 12:59:46.413271 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.235? Tell 192.168.1.2 63 12:59:46.413271 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.236? Tell 192.168.1.2 64 12:59:46.413272 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.29? Tell 192.168.1.2 65 12:59:46.413272 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.31? Tell 192.168.1.2 66 12:59:46.413272 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 67 12:59:46.413273 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.38? Tell 192.168.1.2 68 12:59:46.413273 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.44? Tell 192.168.1.2 69 12:59:46.413273 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.70? Tell 192.168.1.2 80 12:59:48.366589 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.95? Tell 192.168.1.3 81 12:59:49.110848 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.95? Tell 192.168.1.3 83 12:59:50.110864 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.95? Tell 192.168.1.3 146 12:59:52.110584 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.43? Tell 192.168.1.3 177 12:59:54.866488 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.43? Tell 192.168.1.3 179 12:59:55.612297 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.43? Tell 192.168.1.3 181 12:59:56.610828 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.43? Tell 192.168.1.3 258 12:59:58.369865 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 298 12:59:59.110496 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 300 13:00:00.110756 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 302 13:00:01.366468 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 304 13:00:02.110791 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 305 13:00:02.213143 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.30? Tell 192.168.1.3 312 13:00:03.111644 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.143? Tell 192.168.1.3 388 13:00:04.872719 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.41? Tell 192.168.1.3 389 13:00:04.872720 FujitsuT_67:10:6c Broadcast ARP 60 Who has 192.168.1.89? Tell 192.168.1.3 the list is alot longer and I can provide more infomation on request. my only guess is this isn't normal behaviour. 192.168.1.2 and 192.168.1.3 are both of our servers. it's half term so all other workstations are off except from a few teachers left on from friday and I've not noticed them yet but I've just RDP's to them to shut them down. I ran wireshark from my client machine in the staffroom, What steps can I take to work out what is causing the excessive broadcasts? Thanks Bryan
ChrisH Posted October 24, 2011 Posted October 24, 2011 Is 192.168.1.3 a server or a client running some kind of classroom management software?
RageRiot Posted October 24, 2011 Posted October 24, 2011 ..1.3 is our 2008 64bit server which run AD, DNS, and network shares, I do have spiceworks running on it but figured that since ..1.2 is broadcasting equally as much it wouldn't be that.
m25man Posted October 24, 2011 Posted October 24, 2011 Your servers 192.168.1.2 and 192.168.1.3 will only send a broadcast or ARP request when DNS or Host lookup fails. If you have excessive broadcasts being made from your servers I would look examine your DHCP/DNS configs carefully as these should be negating the need to broadcast. Most of the time it comes back to a simple typo or missed box in the configuration that causes this. Stale reverse DNS data and scavenging issues giving multiple host references especially. Spiceworks has a discovery process that uses ARP to determine if you have DNS issues and reports as such if there are problems so it could well be linked but should not be enough to be considered a storm.
CyberNerd Posted October 24, 2011 Posted October 24, 2011 Your servers 192.168.1.2 and 192.168.1.3 will only send a broadcast or ARP request when DNS or Host lookup fails. ?? Are you sure about this? ARP maps layer 2 to layer 3. How does anything on the network know the mac to IP translation of anything else without doing an ARP broadcast. I'd say it was perfectly normal behavior, esp as it is the servers performing the broadcast. @RageRiot - whats teh timescale of your wireshark snapshot?
RageRiot Posted October 24, 2011 Posted October 24, 2011 its constant. it goes through a loop it seems. i set wireshark to filter arp and its a constant stream.
RageRiot Posted October 24, 2011 Posted October 24, 2011 i can post alot more lines from the capture i just selected a bunch of them to get the point across.
CyberNerd Posted October 24, 2011 Posted October 24, 2011 (edited) its constant. it goes through a loop it seems. i set wireshark to filter arp and its a constant stream. Edit: sorry I only just looked that the wireshark timestamp is on your post. It does look like excessive traffic - How big is the netwok? is it segregated with VLAN? Edited October 24, 2011 by CyberNerd
CyberNerd Posted October 24, 2011 Posted October 24, 2011 8 12:59:44.742813 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 19 12:59:45.413318 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 66 12:59:46.413272 IntelCor_0b:b3:a4 Broadcast ARP 60 Who has 192.168.1.33? Tell 192.168.1.2 This doesn't look good at all, all to the same address in a fraction of a second - have you correctly enabled STP on your switches? I'm not sure if this could be a sign of an ARP flood attack?
m25man Posted October 24, 2011 Posted October 24, 2011 ?? Are you sure about this? ARP maps layer 2 to layer 3. How does anything on the network know the mac to IP translation of anything else without doing an ARP broadcast. I'd say it was perfectly normal behavior, esp as it is the servers performing the broadcast. @RageRiot - whats teh timescale of your wireshark snapshot? Maybe I didnt explain that well enough, @CyberNerd is quite correct, but the servers build and maintain their own ARP cache. The ARP cache contains IP address / MAC address translations so that every time an IP packet are to be sent, the MAC address doesn't have to queried through a broadcast, instead it can use the cached address. Windows name resolution is always DNS > Hosts File > Broadcast If the first two fail or are unavailable the default is to Broadcast, therefore when DNS is in trouble the first thing to go through the roof is broadcast traffic. Clients on the network request resolution of network objects by making DNS requests to the server, the server would normally return valid DNS data but if the information is unavailable or incorrect the server will in turn make the appropriate ARP in the hope for a response. This is usually accompanied by hundreds of clients doing the same thing as they start trying to resolve MAC to IP and IP to Netbios. ARP requests are "Normal" on any server just to maintain the MAC to IP tables but when they reach this level its normally a sign there are DNS issues elsewhere. It need not even be the server itself, I have seen devices with bad network information, wrong DNS, Gateway or subnet mask settings all result in the inability to resolve a host correctly which in turn propagates the broadcast traffic. Fortunately for me its a 2 minute job as I just plug in the Etherscope and a few minutes later it will tell me which device has the wrong settings....
m25man Posted October 24, 2011 Posted October 24, 2011 You might want to try a tool like this Monitors Windows ARP cache and allows to send ARP request Run a scan and look for the "Invalid" message in the type column, this can reveal hosts or devices that are missing/off-line or just badly configured and any of those can be at the root of your problem.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now