SWITCHES (ORDER OF THE BOOT)

[Koldov]

No, it's not a Monty Python sketch...

 

Should the order of switches booting up from say a power-cut, (definitely not from me turning them off to move the UPS into the cab, oh no) matter...?

 

I have zero networking knowledge so I'm not sure 'exactly' how ours are configured, but I will try to explain and what I think happened in lay-man's terms.

 

We have:

 

1 x Cisco 5508 WLC

 

1 x Cisco 3750 PoE (labelled Member 1) - This seems to be dedicated to the WAPs

 

1 x Cisco 3750 (labelled Master 2) - This seems to be dedicated to the wired network

 

The 2 x Cisco 3750s are 'joined' I believe (?) with a 'Stackwise' cable.

 

After the erm... power loss today, I got a call saying the Wi-Fi didn't seem to be working. Although I watched everything come back up and all the lights were on...

 

I went through all my paperwork, network diagrams, spreadsheets.... did all the pings and tried to connect to the 5508 WLC web page but couldn't!

 

So, back to the switch room and verified all the lights were on and as it was 'obviously' Wi-Fi related, I then did a reboot of the 5508 WLC.... nothing changed. Checked all the cables to see if I'd dislodged anything, but nothing was obviously wrong.

 

However... looking at the other switches, the 3750 that was labelled 'Member' had a tiny little light on the 'mode' display next to 'Master'.

 

Well, this had to be the problem! So I turned it off and rebooted the other 3750 which then lit up the 'Master' light, so I booted the other 3750 (Member) which obviously didn't get the 'Master' light aned finally rebooted the 5508 WLC...

 

A few minutes later the Wi-Fi came back up and everything was back to normal!

 

I mean it's not world-ending stuff and as long as I remember next time that this is what I have to do it doesn't 'really' matter, but the question is (bearing in mind my limited networking skills) is there anything simple I can do to stop this happening in future?

 

If anyone feels like giving a more technical explanation (and obviously feel free to go wild with suggestions of re-stacking/splitting/fail-over solutions) just bear in mind I won't be doing anything like that...

 

Edit: Probably should have posted in Wired Networks' but initially when I started this thread I though it was just a 'wireless' issue...

Edited January 29, 2024 by Koldov

[Davit2005]

Can you see anything in DHCP logs to say the APs were contacting the DHCP server, does the WLC show any logs of what was going on?

 

3750 acting as a logical stack it should not matter, have you got the WLC connected to both 3750's, link aggregation?

 

Must admit not familiar with the WLC though.

[pete]

I'd be looking to review the config of the stack and the WLC just to see if something's odd / off.

 

What happened to Master 1? And (if it's gone) have any references to it in the stack config been removed?

[PotNoodleTech]

It shouldn't make a difference as it's almost impossible to have a boot order when recovering from a power cut for example.

 

Unless your switches are DHCP and relying on some other server or device for that. Personally I always set everything to do with infrastructure to a static IP address so it never needs to rely on anything else to boot and connect.

[Disease]

It shouldn't make a difference as it's almost impossible to have a boot order when recovering from a power cut for example.

 

Unless your switches are DHCP and relying on some other server or device for that. Personally I always set everything to do with infrastructure to a static IP address so it never needs to rely on anything else to boot and connect.

 

Same here, all network and Servers are statics.

[Koldov]

Can you see anything in DHCP logs to say the APs were contacting the DHCP server, does the WLC show any logs of what was going on?

 

Unfortunately here the DHCP is run from the switches! I know it's unusual, but it's never caused any major headaches that I know of, but it does mean I'm not really skilled enough as Cicio ios and command line to go through the logs...

 

3750 acting as a logical stack it should not matter, have you got the WLC connected to both 3750's, link aggregation?.

 

Sorry, not really got any idea. Following the cables from the WLC, it appears there are 2 going to the 3750 (Member 1) which deals with the WAPs (so at least that makes some kind of sense to me).

 

I'd be looking to review the config of the stack and the WLC just to see if something's odd / off.

 

What happened to Master 1? And (if it's gone) have any references to it in the stack config been removed?

 

I don't know know what happened to Master 1! We used to run the network for the school next to us (long story), but since that stopped I've only pulled their WLC out of the cab. I always thought it was just a labelling/config error and Member has got '1' and Master got '2'...? I'm not a network engineer and I didn't put the stuff in, so why thing got labelled as they did, I have no idea... It does bug me, but I've always ignored it as I'm not sure if it's relevant, we have other switches (Cisco 2960s) and one of those is '02' and another '08' but there certainly isn't 03 - 07 anywhere in the building.

 

It shouldn't make a difference as it's almost impossible to have a boot order when recovering from a power cut for example.

 

Unless your switches are DHCP and relying on some other server or device for that. Personally I always set everything to do with infrastructure to a static IP address so it never needs to rely on anything else to boot and connect.

 

I can't imaging they're set to DHCP (especially considering they are Gateways/DHCP servers themselves) although they could be 'preferred' somewhere they've never changed IP. We've also had the whole network re-IP'd and re-VLan'd a few years ago when changing to LGfL so I'd image all stray bits in the configs were tidied up then, I certainly don't see anything

[PotNoodleTech]

I can't imaging they're set to DHCP (especially considering they are Gateways/DHCP servers themselves) although they could be 'preferred' somewhere they've never changed IP. We've also had the whole network re-IP'd and re-VLan'd a few years ago when changing to LGfL so I'd image all stray bits in the configs were tidied up then, I certainly don't see anything

 

Well preferred = dhcp so that could cause some problems if the switches are rebooting and the switches deal with dhcp!

[Koldov]

I think maybe it all started from the reboot of the cab and that the 3750 'Member' booted first and upon not sensing the master decided to take that title for itself... I guess if the master is no longer the master something along the line breaks down and the rest don't get whatever they need...

 

Just clutching at straws because I don't know if that's the case!

Edited January 29, 2024 by Koldov

[Koldov]

Well preferred = dhcp so that could cause some problems if the switches are rebooting and the switches deal with dhcp!

 

Haha, yes of course but I'm not saying the switches are DHCP (I'm not even sure how to check), just that because they never change and always come up with the same IP doesn't necessarily mean they're static was all.

[psydii]

When cold-starting a system (particularly if you are unfamiliar with it), generally start from the lowest point/core and work your way up/out.

 

With networking, that typically would be the switch stack that is connected to servers / firewall and all the other switches. Then bring up the DHCP and DNS Servers. Then bring up the network management platform, then bring up the rest of the switches working outwards from the core.

 

If any of the core has PoE or the edge is started before the core, you may find devices connected to these ports have started before the necessary network services (DHCP/DNS/Management plane) so you might need to power cycle them again.

 

For some wireless systems, restarting while the switched network has not fully started will result in them going into a wireless mesh, or split brain (where some are in one management domain and some are in another), or worse they start doing very odd things with trunked ports and get their switch ports shutdown by Spanning Tree. Disabling PoE for the wireless switch ports (but not saving the config), then rebooting the edge switch (which clears spanning tree decisions and reloads the last saved config) will usually clear the problem.

 

We have the extra fun problem that there are still inline power injectors for some devices, so we can't easily cut power, and they get properly grumpy if the switch isn't up for a few minutes, we have to go messing around in cabinets and ceiling voids to bring these back on line.

[Chuckster]

If they're stacked switches and configured correctly, then it shouldn't matter which switch powers on first. Switches in a stack elect a single commander whilst the rest are standby and are members. Whichever one powers on first becomes the Commander based on its priority (higher priority number means you want a particular switch to be the Commander at all times)

 

I would advise you to look at your config and revise. It could well be that it's stacked using the appropriate cables but not configured to be a logical switch.

[Koldov]

Thanks for all the answers, I'll dig into it more when I have the time.

 

However, for the last couple of hours I've been firefighting staff members telling me they have no internet (internal network is fine, they can get to the servers, just not Google etc. so they can't teach).

 

Nothing really to narrow it down as such, it's a bit random.... wired computers, wireless laptops, static and DHCP... I don't know where to start!!!

[Oaktech]

You may have a 'commit'ment issue?

 

Worth noting that for some reason one of my sites has an issue where if you lose contact with DHCP. the DHCP server needs to be restarted before the Wifi starts being able to do DHCP stuff again. This may be a DHCP server issue as much as anything else, because the issue for us has persisted across a change from Meru wireless and HP switching through to all Unifi wifi and switching.

 

Might be worth a look though.

[Chuckster]

Is your DNS forwarders looking fine?

From your core switch which gateway is it pointing to?

 

Ususally this is your firewall for both.

[Koldov]

If they're stacked switches and configured correctly, then it shouldn't matter which switch powers on first. Switches in a stack elect a single commander whilst the rest are standby and are members. Whichever one powers on first becomes the Commander based on its priority (higher priority number means you want a particular switch to be the Commander at all times)

 

I would advise you to look at your config and revise. It could well be that it's stacked using the appropriate cables but not configured to be a logical switch.

 

Yeah I did see this in my frantic research but didn't give it a thorough read as I was in panic mode...

 

I didn't really understand though because even if a switch has a low priority, when it comes up first and doesn't detect any other switches, if it has a little vote, does it not say 'even though I'm a low priority, I'm the only switch so I should be master'...

 

Or will it just wait 'forever' for a higher priority switch to throw its hat into the ring...?

Edited January 29, 2024 by Koldov

[Koldov]

Is your DNS forwarders looking fine?

From your core switch which gateway is it pointing to?

 

Ususally this is your firewall for both.

 

DNS forwarders are the LGfL DNS servers I think (172.30.178.53 & 172.30.178.54)

 

Not sure how to check where the Core Switch is pointing...

[Chuckster]

Yeah I did see this in my frantic research but didn't give it a thorough read as I was in panic mode...

 

I didn't really understand though because even if a switch has a low priority, when it comes up first and doesn't detect any other switches, if it has a little vote, does it not say 'even though I'm a low priority, I'm the only switch so I should be master'...

 

Or will it just wait 'forever' for a higher priority switch to throw its hat into the ring...?

 

 

 

You're correct in what you have said. And when the higher priority switch comes online, it will not assume the role as the commander/master. The only time the roles will change again is on the next reboot.

[Chuckster]

DNS forwarders are the LGfL DNS servers I think (172.30.178.53 & 172.30.178.54)

 

Not sure how to check where the Core Switch is pointing...

 

SSH into your switch and type 'show running-configuration'

 

You will see something in the lines regarding IP Route, which ought to be pointing to your onsite firewall provided by LGfL.

 

If there's anything about DNS, then they should be pointing to your DNS servers.