Rebooting Meru 832AP's bring down entire network (wired and wireless)

[JHatz]

I have an interesting situation since I've put Meru 832e's in our high school. Whenever I reboot three or more - my entire network - goes down. If I reboot just one - I lose network connectivity district wide for probably ten seconds (our pings come back as unreachable) then normal network resumes. When rebooting three or more - maybe one ping in 20 get to their destination - if lucky.

When I say entire network - I mean it - wired, wireless - internal and external communications - all gone.

 

To "fix" this - in some cases I can just unplug the ethernet cable from the Meru 3200 controller, wait a few seconds and plug back in - usually network goes back to normal after fifteen to twenty seconds. In some cases I have to reboot the Meru controller. I can replicate this anytime by rebooting three AP's.

 

Anyone else ever have a similar experience? I am starting to believe it may be ARP poisoning.. we've run wireshark against the controllers swith port and don't seen any flooding going on, nor anything in the pcaps that jump out at us. Our network switch is Enterasys - just over a year old. The Ap's were installed this summer and I didn't reboot any AP's until last month when I was upgrading them and the controller.

[Driftingashore]

I'm certainly no expert, but if there were a broadcast/multicast storm going on that's big enough to bring down your entire network it shouldn't really be hard to find it using Wireshark as those packets would be flooding everywhere (assuming you're monitoring on the same VLAN/Subnet as the traffic). If that's not the case, I'd try mirroring all ports on a switch that's connected to a few access points to a specific port with a client running Wireshark, rather than just focussing on the controller, and also make sure that any firmware on the controllers / APs is up to date. Once you have a bigger picture in Wireshark it should give you a much better idea of what is going on. It may also be worth trying to see if this happens on all APs or just ones connected to/through a certain switch, and likewise if you're able to ping another device on the same switch as yourself, or if it's a certain route / bottleneck getting overloaded and causing drops.

 

I'd be at least a little suspicious of the fix simply as it's one of those "do something and wait for a bit" fixes that may just be a case of the issue resolving itself in that time. I don't really have any answers though. Good luck!

[computer_expert]

I know meraki has a feature where they can create a mesh based network with the next nearest AP(s) in the case of switch/cable etc failure. Not sure if Meru have a similar feature but it may be worth checking out.