twin--turbo Posted January 14, 2016 Posted January 14, 2016 Switches all have loop protection (those that are manageable) turned on, so it shouldn't be happening... Un-managed switches, generally have no control over loop protection. As I mentioned previously disconnect all other switches from the new core and see if it improves. If it does not then take out other cables int bunches and narrow down where the fault to a port(s) TT
RobD Posted January 15, 2016 Posted January 15, 2016 In my last role in a School all the HP pro switches didn't have have spanning tree on as standard but if its intermittent and only affects certain PC's I doubt its flooding of the network. You could mirror the port, plug in a laptop with wireshark, do a capture and save it and I could have a go at looking at it but I'm definitely not a expert. I've been doing my best to learn about the fundamentals of wireshark from here: PacketBomb – Packet Analysis Explosion
FN-GM Posted January 15, 2016 Posted January 15, 2016 Still no idea what's going on here. Feels like a loop, as I've had a similar experience in my previous workplace with the same symptoms. I've tried analysing with WireShark, but everything looks normal? Nothing sending huge bytes etc. (not that I'm overly familiar with WireShark). Switches all have loop protection (those that are manageable) turned on, so it shouldn't be happening... It's across the entire site, but not everyone is affected. If its a loop the packets wont be large, you will have loads and loads of broadcasts.
mikeyd101 Posted January 15, 2016 Posted January 15, 2016 (edited) If its an intermittent problem you might not be looking at the same time its happening. - Do periodic scans during different times of day / longer scans. - Ensure your plugged into the core switch and your seeing traffic from everywhere. - as FN-GN said your looking at vast quantities (tens of thousands) of packets generated not the size. and to the broadcast address something like 192.168.0.255. Broadcast address is the last address in the ip block range can be tricky to work out if you have weird ip/subnet addresses. We had a similar problem with network slow and crashing, turn out to be a bad NIC in a machine. Edited January 15, 2016 by mikeyd101
GRitchie Posted January 18, 2016 Author Posted January 18, 2016 Okay. Talk to me guys! I'm in the server room now unplugging things. I've got my laptop (this one) plugged DIRECTLY into the core switch. I'm getting ping responses of 1, <1 and then anything from 30-130. Primarily it's <1 and 1. Unplugging things in turn isn't having an effect. Wireshark: Conversations view, IPv4, sort by packets: Lots coming from an IP address that resolves to a computer which isn't even part of the network anymore (old isolation machine) Second top is from the data server. Fire away. Because I have no idea
psydii Posted January 18, 2016 Posted January 18, 2016 What sort of traffic is that 'off net' pc throwing out?
GRitchie Posted January 18, 2016 Author Posted January 18, 2016 UPDATE - found the PC, IP has been moved for an IP Camera. Took the IP camera off the network, was still flooding even though it wasn't plugged in??? See below, screenshot of the conversations view (top few anyway) 16.3 is the smoothwall box
psydii Posted January 18, 2016 Posted January 18, 2016 Can we get a view of the packets it was sending? (if it continued after you unplugged it, then it wasn't the only thing on your network with that IP)
FN-GM Posted January 18, 2016 Posted January 18, 2016 (edited) UPDATE - found the PC, IP has been moved for an IP Camera. Took the IP camera off the network, was still flooding even though it wasn't plugged in??? See below, screenshot of the conversations view (top few anyway) 16.3 is the smoothwall box [ATTACH=CONFIG]34434[/ATTACH] Whats your network design like? Seems like there is a flaw in it somewhere. Enable flood protection on your switches. If you get things like this it will automatically disable the port the problem device is connected to. Edited January 18, 2016 by FN-GM
pantscat Posted January 19, 2016 Posted January 19, 2016 Do you have the mac address of the device that is flooding the network? If you've got Procurve switches you can use the "show mac-address xx:xx:xx:xx:xx:xx" command to see what port the device is connected to...
mikeyd101 Posted January 19, 2016 Posted January 19, 2016 UPDATE - found the PC, IP has been moved for an IP Camera. Took the IP camera off the network, was still flooding even though it wasn't plugged in??? See below, screenshot of the conversations view (top few anyway) 16.3 is the smoothwall box [ATTACH=CONFIG]34434[/ATTACH] Conversations view (point 2 point) will only show back and forth connections, excluding things like broadcasts which are more likely to be traffic storming. Look at endpoints (which will show broadcasts as a separate endpoint) and/or IO Graphs (for overview of amount of traffic going through the switch) also statistics - summary - avg packets/sec, avg packet size, bytes and avg bytes/sec. I wouldn't be concerned with packet counts that low (the 1 highlighted 244 packets / 32k of data) even the top 1 with 15k packets /14mb i wouldn't be overly concerned about over that duration (176 sec / 3 minutes).
twin--turbo Posted January 19, 2016 Posted January 19, 2016 Are you mirroring the traffic to the port the wireshark laptop is plugged into?
ricki Posted January 20, 2016 Posted January 20, 2016 My best advice is make a plan an check the following one at a time Check tree spanning on all switches Plug server into core switch and pull out all other switches does a client plugged into core work OK Add feed switches then cabinets one at a time checking client ok Check dns and replication of domain controllers Check event viewer Check time and date synced with server Try disabling on server nice cards Chimney Ross Check sum ip4 If you don't use ip6 try disabling it Richard
ricki Posted January 20, 2016 Posted January 20, 2016 Sorry I forgot one do one thing at a time Richard
GRitchie Posted January 20, 2016 Author Posted January 20, 2016 Morning. I ended up having a company remotely take a look (the company who provided the new switch). Simply upgrading the firmware on it seems to have massively helped the issue. Ping responses are still not perfect, but they're not going above 10 (when plugged directly into the core switch). Anyone any ideas why this may have been?
Steve21 Posted January 20, 2016 Posted January 20, 2016 I'd want to give it a few days before blaming that as the direct cause, as obviously doing the update normally restarts the switch anyway, so anything like loops would drop off. Depending on how old the firmware was originally there were a few fixes around the CPU of the switch idling high most the time, as it's not really designed for core etc might be more apparent if lots going into it. (Compared to an edge with only a few devices in etc) But guess time will tell Steve
GRitchie Posted January 20, 2016 Author Posted January 20, 2016 I suppose so! As I said, things still aren't perfect. But they're a lot quicker. What isn't good, is there are lots of little mini hub/switches dotted around the network (everywhere). Which really won't be helping matters. That, and the managed switches that are in use are primarily Netgear FSM726 or equivalents.
Steve21 Posted January 20, 2016 Posted January 20, 2016 Do you know the CLI for HP? As you can monitor things like the CPU on switches with it, so might be worth doing that and seeing how it's running. Support - FAQs - HP ProCurve Networking Steve
ricki Posted January 20, 2016 Posted January 20, 2016 I would also up date the firmware on your other switches and check for loops on them 1
twin--turbo Posted January 20, 2016 Posted January 20, 2016 I suppose so! As I said, things still aren't perfect. But they're a lot quicker. What isn't good, is there are lots of little mini hub/switches dotted around the network (everywhere). Which really won't be helping matters. That, and the managed switches that are in use are primarily Netgear FSM726 or equivalents. Did you say the "H" word. Really??? It reminded me of something I have overlooked and could well be a big contributor. Speed/Duplex Mismatch can easily cause the symptoms your seeing. Set all ports on the new switch to 100/Auto and see if it improves. TT
twin--turbo Posted January 20, 2016 Posted January 20, 2016 Any hubs will be HDX, if they really exist! TT
BigDF Posted January 20, 2016 Posted January 20, 2016 You should plug directly into the core switch and run a command to view the port utilisation and cpu utilisation. If any ports are running high, (best to zero the results then wait a few seconds). Trace the high ports back to the edge switch or device they relate to and run the same test on that device and so on until you reach the very end device. If the utilisation stats show all is normal it might not be a loop, they do tend to just bring everything to a stop. Any dumb switches on the network at all?
twin--turbo Posted January 20, 2016 Posted January 20, 2016 (edited) You should plug directly into the core switch and run a command to view the port utilisation and cpu utilisation. If any ports are running high, (best to zero the results then wait a few seconds). Trace the high ports back to the edge switch or device they relate to and run the same test on that device and so on until you reach the very end device. If the utilisation stats show all is normal it might not be a loop, they do tend to just bring everything to a stop. Any dumb switches on the network at all? whilst your on the console (this is the serial connection to the switch) turn on console logging. Try all first all though that may be a bit verbose (none|debug|all|not-info|critical) #console events all Remember to turn it off afterwards #no console events Edited January 20, 2016 by twin--turbo
GRitchie Posted January 21, 2016 Author Posted January 21, 2016 @twin--turbo thanks for your comments. The support company I've been working with also recommended setting the ports with hubs to 100-auto. I've tried, but thing still aren't perfect. If I could just burn them all I would, but unfortunately even computer classrooms are networked with them :-|
GRitchie Posted January 21, 2016 Author Posted January 21, 2016 Replaced another switch last night, seems to be helping me to fault find at least. Take a look at this... :'(
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now