AngryTechnician Posted July 24, 2012 Posted July 24, 2012 (edited) Last weekend I finally got around to setting up Layer 3 routing on my core switch (a ProCurve 5308xl) and re-addressed the entire network to split it up into sensible(?) VLANs, with 1 subnet per VLAN, and the switch routing traffic between them. Since then, everything works perfectly about 95% of the time: clients pick up DHCP addresses, everything uses the switch as the default gateway, and traffic passes to and fro normally. Then every so often (once an hour or so?), something inexplicable happens. Basically, the switch will suddenly decide it cannot route to some (but not all) addresses from one VLAN to another. Example 1: I'm sitting on a client on VLAN 6, and I'm browsing BBC News (via our gateway router on VLAN 7). Then I go to EduGeek and find it won't load. I start a PING, and this is what I see: Pinging www.edugeek.net [176.9.42.234] with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Request timed out. Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=40ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=41ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=40ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Reply from 176.9.42.234: bytes=32 time=39ms TTL=55 Basically, after less than a minute of PING, the remote address will start responding and everything works again. It always comes back quite quickly; so far I've never had a delay of more than about a minute. It will then be fine again for some time - hours or more - before the exact same thing happens again, maybe with the same address, maybe with a different one. It's like it just forgets how to reach an address, then after nagging it for a minute or two, it suddenly remembers and starts passing traffic. This also happens (less frequently) with LAN addresses, so it's not just Internet routing that is affected. Basically, anything that has to cross to a different VLAN or be routed out by the switch seems to be at risk, but not in any pattern I can identify. Here's what my investigations have uncovered so far: At all times when this is happening, I can initiate a PING from the core switch via a telnet console session, and the affected address responds to the switch instantly, while still not responding to the client. I can always PING other addresses in the same subnet/VLAN as the client, including the switch, which is acting as the default gateway for each subnet. It seems to be only individual addresses that are affected at any one time, rather than whole VLANs or subnets: I've had instances where I can't ping 192.168.2.1, but I can ping 192.168.2.2. When an address becomes unreachable, it becomes unreachable for ALL clients on all all VLANs (including the default VLAN) simultaneously, except for the core switch (which can always reach everything). I can't tell if it literally only affects a single IP at a time. I don't think I've had a situation yet where multiple addresses are affected, but it lasts for such a short time that it's tricky to do much testing before the problem disappears again. After 2 days of poring over forums and switch config, I am almost at my wit's end. This switch has been in use without L3 without any issues, suggesting its something to do with my config, but I can't work out what, especially as it works as designed most of the time. Here is the config I'm using: ; J4819A Configuration Editor; Created on release #E.11.21 hostname "Core" max-vlans 16 time daylight-time-rule Western-Europe module 7 type J4821A module 8 type J4821A module 6 type J4878A module 4 type J4820A module 3 type J4820A module 1 type J4820A module 2 type J4820A module 5 type J4878A interface H1 no lacp exit interface H2 no lacp exit interface H3 no lacp exit interface H4 no lacp exit trunk H1-H4 Trk1 LACP sntp server 192.168.3.1 ip routing ip udp-bcast-forward timesync sntp sntp unicast snmp-server community "public" Unrestricted vlan 1 name "DEFAULT_VLAN" untagged A1-A5,A7-A24,B1-B5,B7-B10,B12-B15,B17-B24,C3-C24,D1-D21,D23,E1-E4,F1 -F4,G2-G3,Trk1 ip address 192.168.11.254 255.255.252.0 ip helper-address 192.168.3.22 no untagged A6,B6,B11,B16,C1-C2,D22,D24,G1,G4 ip igmp exit vlan 50 name "iSCSI" no ip address tagged F2,Trk1 exit vlan 100 name "NM" ip address 192.168.100.254 255.255.255.0 ip helper-address 192.168.3.22 tagged E1-E4,F1-F4,G3,Trk1 ip igmp exit vlan 200 name "Backup WAN" untagged D24 no ip address tagged F4 exit vlan 2 name "Networking" untagged C1-C2 ip address 192.168.2.254 255.255.255.0 ip helper-address 192.168.3.22 tagged A15,E1-E4,F1-F4,G3,Trk1 exit vlan 3 name "Servers" untagged B6,D22 ip address 192.168.3.254 255.255.255.0 ip helper-address 192.168.3.22 ip forward-protocol udp 192.168.11.255 7 tagged F2,Trk1 ip igmp exit vlan 6 name "IT Support" untagged G4 ip address 192.168.6.254 255.255.255.0 ip helper-address 192.168.3.22 ip forward-protocol udp 192.168.11.255 8992 tagged E1-E4,F1-F4 ip igmp exit vlan 4 name "Printers" untagged A6,B11 ip address 192.168.4.254 255.255.255.0 ip helper-address 192.168.3.22 tagged E1-E4,F1-F4 exit vlan 7 name "Perimeter" untagged B16,G1 ip address 192.168.7.254 255.255.255.0 exit ip route 0.0.0.0 0.0.0.0 192.168.7.1 spanning-tree spanning-tree Trk1 priority 4 ip multicast-routing router pim exit vlan 1 ip pim all exit vlan 6 ip pim all exit password manager Apart from removing a couple of identifying lines (the snmp-server contact and snmp-server location lines), this is the entire config. As you can see, the VLANs are entirely port-based at the moment, with no ACLs - all of that is coming later. The VLANs that don't have IP addresses configured on the switch are meant to be that way, as there are some VLANs I don't want to route. I've eliminated the multicast routing/PIM as a cause, as the problem happens even with that section removed. Oh, and yes, VLAN 1 is meant to have a 192.168.8.0/22 range rather than /24. I know it looks odd but it's meant to be that way, and again, the problem still occurs if it's a /24 range. Here's the routing table, which to me looks as expected: IP Route Entries Destination Gateway VLAN Type Sub-Type Metric Dist. ------------------ --------------- ---- --------- ---------- ---------- ----- 0.0.0.0/0 192.168.7.1 7 static 1 1 127.0.0.0/8 reject static 0 250 127.0.0.1/32 lo0 connected 0 0 192.168.2.0/24 Networking 2 connected 0 0 192.168.3.0/24 Servers 3 connected 0 0 192.168.4.0/24 Printers 4 connected 0 0 192.168.6.0/24 IT Support 6 connected 0 0 192.168.7.0/24 Perimeter 7 connected 0 0 192.168.8.0/22 DEFAULT_VLAN 1 connected 0 0 192.168.100.0/24 NM 100 connected 0 0 CPU and memory usage seems fine: typically more than 50% free memory, and CPU fluctuates between 2-40%, but not seen it go higher. I'm about a day away from calling in a consultant at this point, which will make me extremely grumpy. If anyone here can give me a clue, it would be very very welcome. Edited July 24, 2012 by AngryTechnician
Michael Posted July 24, 2012 Posted July 24, 2012 I can't see anything wrong with your config at first look. Have you tried updating the firmware and/or resetting the switch?
AngryTechnician Posted July 24, 2012 Author Posted July 24, 2012 Firmware is the latest (E.11.21). Have done a reload a couple of times now and the problem remains.
CyberNerd Posted July 24, 2012 Posted July 24, 2012 (edited) I've seen similar problems which turned out to be a loop somewhere. I don't know the commands on procurves, but basically what you should do is: 1) From the core switch, ping something that occasionally drops off the network. 2) Use a command on the procurve to determine mac address (display arp) in comware. 3) Then find the port or aggregation group that is showing that address (display mac-address xxxx) in comware will show the port associated with that mac address 4) Now ping offending device when it goes offline 5) Again do the equivalent of "display mac-address xxxxx” to determine port or aggregation group when the problem device is offline. Make a note if the port or aggregation group changes. This will show you the ports/aggregation groups that are causing issues. 6) Repeat on the other switches to determine loop. 7) Fix the loop or shutdown port edit: xxxx is the mac address shown in display arp - not the ip address. Edited July 24, 2012 by CyberNerd 2
DMcCoy Posted July 24, 2012 Posted July 24, 2012 Have a play with mac-age-time try something like 600 seconds. 1
Abaddon Posted July 24, 2012 Posted July 24, 2012 I've GOT a consultant in at the moment - I'll pass this over his desk in the morning and see if he can spot anything obvious(to him).. He is extremely good, and I've had him in every year to help out with some switch configuration or other. He's doing some stuff that means he probably has time for this, so I'll let you know if he has any ideas..
AngryTechnician Posted July 24, 2012 Author Posted July 24, 2012 @CyberNerd: A loop had crossed by mind already - wouldn't STP catch that? I have it enabled on all switches. And wouldn't I have seen problems before enabling L3 routing? @DMcCoy: Will have a look at mac-age-time tomorrow, thanks. @Abaddon: Hopefully he won't charge you extra for looking at someone else's problems, but thanks!
Jona Posted July 24, 2012 Posted July 24, 2012 If your losing connectivity for between 60 and 30 seconds it's often because Spanning Tree is setting a port to the blocked state as it's detecting a loop somewhere. Depending on how your vlan ports, trunks, etc are set-up this appear to be layer 3 issues when it's actually a layer 2. When a machine is in a failure state can it ping it's gateway ip on the switch? Can other places? Have you checked your switch logs? Usually command: show log on a procurve.
Destinova Posted July 25, 2012 Posted July 25, 2012 I'd also suspect STP but noticing you have some iSCSI traffic on there as well I'd check for multicast issues.
CyberNerd Posted July 25, 2012 Posted July 25, 2012 @CyberNerd: A loop had crossed by mind already - wouldn't STP catch that? I have it enabled on all switches. And wouldn't I have seen problems before enabling L3 routing? As @Destinova and @Jona said - it could be STP causing the issue. The procedure that I proposed would help determine this.
Theblacksheep Posted July 25, 2012 Posted July 25, 2012 (edited) For what its worth we don't use STP on our HP switches (5304's and a 5412) I seem to remember one of the consultants (a while back now) recommended mdix on ports instead of STP for loopback prevention. Edited July 25, 2012 by Theblacksheep
Jona Posted July 25, 2012 Posted July 25, 2012 @Theblacksheep I'm pretty sure auto-mdix won't deal with a loopback properly, it's usually used to support straight vs. crossover cables see: Medium Dependent Interface - Wikipedia, the free encyclopedia It may mitigate to some extent against the scenario where a student plugs two network points together if the two network points are in the same switch. I can't see anyway it can mitigate against an actual switching loop, where you accidentally connect a loop of switches which in a complex network is surprisingly easy to do!
Theblacksheep Posted July 25, 2012 Posted July 25, 2012 @Theblacksheep I'm pretty sure auto-mdix won't deal with a loopback properly, it's usually used to support straight vs. crossover cables see: Medium Dependent Interface - Wikipedia, the free encyclopedia It may mitigate to some extent against the scenario where a student plugs two network points together if the two network points are in the same switch. I can't see anyway it can mitigate against an actual switching loop, where you accidentally connect a loop of switches which in a complex network is surprisingly easy to do! AFIK auto is the default. Sorry, class end-points are set to mdix. Maybe its coincidental we've not had any loopbacks.
AngryTechnician Posted July 25, 2012 Author Posted July 25, 2012 (edited) If your losing connectivity for between 60 and 30 seconds it's often because Spanning Tree is setting a port to the blocked state as it's detecting a loop somewhere. Depending on how your vlan ports, trunks, etc are set-up this appear to be layer 3 issues when it's actually a layer 2. When a machine is in a failure state can it ping it's gateway ip on the switch? Can other places? Have you checked your switch logs? Usually command: show log on a procurve. Thing is, when I lose access to a a website (e.g. EduGeek), I still get access to other websites (e.g. BBC) and I can still ping our Internet gateway, AND the site is still pingable from the switch via telnet. Unless I drastically misunderstand STP, isn't the only way STP could block websites is if it blocked the port the gateway was on, which would affect all websites simultaneously, and even the switch wouldn;t be able to get through? I've checked the logs and they don't indicate any ports being blocked. When a client fails, it can still ping it's gateway IP. I will take a careful look at which ports a MAC address is resolving to when this happens as @CyberNerd suggests, but I'm fairly sure I've looked at this already when websites have been affected and the gateway MAC appears on the same port every time. Will look closely at what happens when it's a LAN address. @Destinova: What sort of multicast issues are you suspicious of? I do have a multicast IPTV server running (though I actually shut it off temporarily last night as I began to scrape the bottom of the barrel for ideas of what the problem could be). When it's running, it's on a separate VLAN to the iSCSI and there is no routing in or out of the iSCSI VLAN. Edited July 25, 2012 by AngryTechnician
SYNACK Posted July 25, 2012 Posted July 25, 2012 Have you tried setting up extended logging for layer3 operations and ACLs. It might be that DNS is getting blocked or that routing is dropping the routes randomly.
AngryTechnician Posted July 25, 2012 Author Posted July 25, 2012 I don't have any ACLs yet, so I don't think I can configure any more logging than what I'm already getting from show logging, can I? I do know that DNS is fine - I can always resolve names to IPs. One notable development: since turning off the IPTV server last night, I haven't had the problem once. Going to leave it off for the rest of the day, especially as I haven't been using my workstation intensively this morning, but we could be on to something here. I already know that the problem was happening before multicast-routing was enabled, but could just having the multicast traffic there in the first place cause something like this? I've had this IPTV box running for 2 years without any issues before turning on L3.
SYNACK Posted July 25, 2012 Posted July 25, 2012 You can turn on full debugging but that will own the switch CPU with debug all more focused you could use debug ip ? to pick out some stuff to do with it This document may be worth a look Diagnostics Guide for the HP ProCurve Routing Switches 9304m, 9308m, and 9315m - Software Release 07.6.04 or Greater (S and ftp://ftp.hp.com/pub/networking/software/9300m-9408sl-Diagnostic-Jun2005-59906032-Chap03.pdf 1
psydii Posted July 25, 2012 Posted July 25, 2012 It's likely that enabling multicast has tipped your core over the edge on a problem it had previously been dealing with without noticable impact. I'd set up a monitor/mirror port on your core and run a packet capture to see if there was anything obvious showing up. <- this is my 'new' favourite trick, I am always reluctant to do it, but it always shows me something I wasn't expecting to see. I wonder whether you've got a loop somewhere and either STP is flapping or Broadcast Suppression is kicking in and the CPUs/Buffers are flooding resulting packet loss. 1
Destinova Posted July 27, 2012 Posted July 27, 2012 (edited) @Destinova: What sort of multicast issues are you suspicious of? I do have a multicast IPTV server running (though I actually shut it off temporarily last night as I began to scrape the bottom of the barrel for ideas of what the problem could be). When it's running, it's on a separate VLAN to the iSCSI and there is no routing in or out of the iSCSI VLAN. I've had issues with IGMP snooping and multicast. If it isn't configured properly across every switch that multicast traffic is going to pass, you'll run into problems. Spanning tree being what it is, unless your topology includes multiple redundant links, I don't know that is where you'd have an issue although you won't see these ports as blocked per se, unless you are actually looking at their STP state. To test the multicast theory you can disable IGMP snooping on all your switches. This is a temporary test and I do not recommend this as a permanent solution but it may point to whether or not this is a multicast problem. A topology if you are able to provide one to us would also help diagnose what the issue might be. Cheers. Edit: Having just read about turning off the IPTV server I'd point at a multicast routing issue. Try and find out what multicast IP that server is using, there may be a conflict with another multicast service. Edited July 27, 2012 by Destinova
AngryTechnician Posted July 27, 2012 Author Posted July 27, 2012 (edited) OK, the multicast server remained off yesterday as I was doing a lab build all day, but there still seem to be no recurrences of the problem. I plan to fire it back up tomorrow when no-one else is in to reproduce the issue and do some more troubleshooting. I know the details of the multicast IPTV server extremely well as it's a personal project. It has been running pretty much 24/7 for 2 years with no issues, so it's a little odd that it's suddenly causing problems now. IGMP is configured on all switches on the VLANs which carry the IPTV: with ~100Mbps of multicast traffic being pumped out by the IPTV, it's very obvious from the traffic graphs if you miss a switch (as I have once in the past). It broadcasts SAP on the normal 224.2.127.254 address, and the ITPV channels are on the following addresses: 239.255.0.1 239.255.0.10 239.255.0.11 239.255.0.12 239.255.0.14 239.255.0.15 239.255.0.18 239.255.0.19 239.255.0.2 239.255.0.21 239.255.0.24 239.255.0.28 239.255.0.3 239.255.0.30 239.255.0.31 239.255.0.38 239.255.0.4 239.255.0.46 239.255.0.47 239.255.0.49 239.255.0.5 239.255.0.50 239.255.0.51 239.255.0.52 239.255.0.54 239.255.0.6 239.255.0.7 239.255.0.70 239.255.0.71 239.255.0.72 239.255.0.80 239.255.0.81 239.255.0.82 239.255.0.85 239.255.0.87 239.255.0.89 239.255.0.9 239.255.1.5 239.255.3.1 239.255.7.1 239.255.7.10 239.255.7.100 239.255.7.11 239.255.7.12 239.255.7.13 239.255.7.14 239.255.7.15 239.255.7.16 239.255.7.18 239.255.7.2 239.255.7.22 239.255.7.23 239.255.7.24 239.255.7.27 239.255.7.28 239.255.7.3 239.255.7.4 239.255.7.5 239.255.7.6 239.255.7.7 239.255.7.8 239.255.7.9 Unless I've missed something, the IPTV streams are all in the administratively scoped multicast address space, and there are no other multicast servers on the network (aside from the WDS server, but I never use the multicast features). I've attached a switch topology graphic. The only time I have more than one connection between any two switches is between the core and the switch immediately below it in the rack for my servers. The 4 links are trunked, and there are no reported errors with that so it is acting as a single 4Gbps link. At this point I'm beginning to wonder if this switch just can't hack that much IGMP/multicast and the routing at the same time, regardless of whether the traffic is to-standard or not. The CPU and memory aren't getting much above 50%, but it wouldn't be the first problem I've had with sub-par performance from HP kit. Edited July 27, 2012 by AngryTechnician
AngryTechnician Posted July 28, 2012 Author Posted July 28, 2012 Right, well I've had a thoroughly frustrating day with this, but I have some answers. It's definitely a problem with the multicast traffic combined with ip multicast-routing. I must have left it on initially when I thought I had ruled it out, because I've been able to literally toggle the problem on and off today by enabling and disabling multicast-routing while having a PING running. The problem occurs even if PIM is disabled, so the multicast traffic isn't even being routed - just the presence of multicast traffic on a single VLAN with routing on is enough. If I have multicast-routing on with no multicast traffic: everything is fine. If I have multicast traffic with no multicast-routing: everything is fine. If I have even the slightest amount of traffic (as little as a single SAP announce) with multicast-routing on, the problem will occur. Another tell-tale symptom I've discovered today is that when the multicast is being routed, the routing performance is poor. SD streams are OK, but on anything but the default VLAN, HD streams are garbled with MPEG decoding artefacts on playback. That doesn't happen on the default VLAN (where the traffic originates), even when testing from the farthest point of the topology over a wireless WDS bridge. There are documented limits for multicast routing on this switch, which get easier to reach the more VLANs you have, and at one point I had thought it might be a load problem due to the amount of multicast traffic. Then when it happened with just the SAP announcements running, which are a tiny amount of text traffic on a single multicast address, I came to the conclusion that the ProCurve 5300 series is just a pile of junk. There are no pertinent messages logged to the debug log when the problem occurs (even with debug all), and there was nothing unusual shown on a monitoring port trace using Wireshark. The switch simply doesn't do what its supposed to. Thanks for all your help and suggestion guys, it has helped me narrow down the issue and preserve my sanity. I will be having an interesting chat next time our local ProCurve specialist calls to make a sale.
SYNACK Posted July 29, 2012 Posted July 29, 2012 Multicast routing is always a one of the more difficult things, I remember when I was doing Cisco stuff and you needed a specific point revision of the right firmware to get it running well with all the streams. Have you tried updating the firmware to the latest version etc.
AngryTechnician Posted July 30, 2012 Author Posted July 30, 2012 Firmware is the latest, last update was in May 2010 so not expecting any further updates. I already have a way around not being able to route the IPTV (the other VLANs will get a unicast stream instead) so I'm not going to lose any more sleep over it, but it would have been nice to know the switch was rubbish before spending a week growing grey hairs trying to figure it out!
CyberNerd Posted July 30, 2012 Posted July 30, 2012 but it would have been nice to know the switch was rubbish before spending a week growing grey hairs trying to figure it out! It has got Procurve written all over it
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now