HELP - Network issues and outside help

[ITGuyNW]

Hi all

 

I'm still very much a newbie network manager and I need some help with an urgent issue.

 

12.30pm people started reporting internet connectivity loss. I thought it was external but BT said everythings normal. We use cambium wifi which have access points that talk directly to the Internet. Basically since 12.30pm everything is dropping in and out, DNS errors, aps up and down and im at a loss. This is also affecting phones which are teams based and registers, which again are through the internet.

 

All servers are up. This is happening across site in every building. An AP will be up one minute, then goes again then comes back up and stays for 20 minutes before going again. Wired connection seems to be ok but browsing can be slow like theres a DNS issue happening.

 

I've checked the domains, smoothwall, done reboots, checked certificates, event viewer and nothing out of the ordinary! Core switches all green lights, everything seems fine! It feels like a DNS or certificate type issue but then again I cant rule out something physical!

 

So I need help and urgently. Is there anyone you can recommend to come in as an extra layer of tech support? Any companies or consultants who could help? I'm based on the Fylde coast.

[speakercon]

Do you have VLANs or a flat network? (Thinking possibly a loop)

 

What happens if you ping 8.8.8.8 -t ?

What happens if you ping your firewall IP -t?

[Oaktech]

I wonder if someone has created a network loop somewhere?

[ITGuyNW]

We have a couple of vlans, one for wifi and one for guest wifi. The rest is flat. Weve not done any updates, configs etc today or this week infact.

 

8.8.8.8 is giving me a reply every 3 or 4 tries. Firewall is doing the same. Mixed results.

[speakercon]

Which VLAN are you pinging from?

 

Could you try pining 8.8.8.8 from each VLAN and see if the results are different?

 

At this point I would suspect a loop or a piece of equipment causing a broadcasting storm - are your switches managed with spanning tree enabled?

 

Any mini switches lurking about?

 

I would try disconnecting each edge cab one at at a time from your core (while keeping 8.8.8.8 pinging) and see if disconnecting one of the edge cabs fixes it and then at least you'll have a rough idea of where the problem is.

Edited January 18, 2024 by speakercon

[simpsonj]

I agree with the others, this sounds like a loop issue (someone plugging a network cable back into the wall, rather than into a device, or possible linking up switches with multiple cables)

 

Worth checking whether the Spanning Tree Protocol enabled on all switches?

 

As well as checking your external connection, it's worth pinging internal devices (Smoothwall, Core Switch, Domain Controller) and seeing if they respond in healthy manner.

 

If possible, you could try disconnecting each edge switch from the core one by one to see if that improves the reliability and isolate the problem. I'm sure they are more sophisticated methods, but this is fairly tried and tested!

[speakercon]

Or a quick walk around your IT suites checking the network ports if you don't have many might reveal the problem - 12.30 - the pupils messing around at lunch perhaps?

 

This is where they usually are in my experience.

[ITGuyNW]

Just seems strange that it would suddenly happen at 12.30.

 

I've not found any devices or anything that would affect this.

 

If I ping the domain controllers from my flat network, replies are fine however if I ping Smoothwall, it gets a ping every 3 or 4 replies. I've not tried from another vlan yet.

 

It just feels like a DNS/certificate type issue.

 

I've had to come home now. I'm going to try and isolate some edge switches tomorrow and run some wireshark tests.

 

As far as I'm aware, spanning tree is enabled throughout, however we never touch switch config or mess with the server room so I don't know why it would suddenly start acting up.

 

- - - Updated - - -

 

Thanks all by the way. Feeling pretty useless at the moment but I hope to find some answers tomorrow.

[TwistedHelixis]

Network up and down does sound like a loop issue, could be something else but looping a network cable out of one network port and back into another is something students love to do.

[TwistedHelixis]

I once had a bizarre issue I really couldn't workout, so I plugged the server, WAN and test PC into a spare switch and started there. Turned out to be a provider issue. Something had slightly damaged the cable somewhere, but for some reason the providers systems wasn't picking up any issues.

[Olliedawg]

From what you have explained it does sound like a high chance of there being a loop somewhere ..

 

I have had a similar issue before, did turn out to be a loop.

 

Have a good look around site, double check IT room network points, ensure none of the network cables are plugged into a network point at both ends..

[HereIGoAgain2601]

We had similar issues with Cambian in one of our schools last summer. Our network at the time was an inherited mess from RM and a lot of incompetent engineers over the years. We spent many days trying to look for loops; in the end it transpired that someone had not excluded the default gateway from DHCP so every time Mr Bloggs connected his mobile phone to the WiFi it took down the whole cambian wifi network That was a fun week.

[MatthewL]

Start at the core check all ok there, if you site is big disconnect the edge switches and start by testing 1 by 1 to narrow your area down then hit that area if the fault happens. Doesn't hurt to reboot switches/routers as I've had some that sometimes just loose their marbles but does sound like a loop issue, hence why any unused ports are never used or disabled.

[Mullaney18]

If you haven't already, bounce your DNS server (if you have one), I would probably get wireshark on the go and see what is causing the most traffic.

 

All while I would start disconnecting sections of your network.

[kevin_lane]

I would be interested to see how things looked with All APs going offline

 

Only reason why is because you said that the APs come on line and then go offline and that hardwired seemed to be ok

 

If that is the case then as you said it could it be a config issue with APs and how they are being controlled

 

Just a thought

[NegativeKillDeath]

Had similar symptoms once. Turned out that a brief powercut over night caused a switch to reboot and revert to a previous config (I swear I had saved it). Anyway some vlan tagging got messed up and traffic was broadcasting across vlans. It took a while for the symptoms to ramp up though.

 

Get a wireshark trace going and see if you can see any suspect traffic.

[ITGuyNW]

So morning update

 

Everything was still in a dead state this morning. Gave Smoothwall a call and the box seemed fine and the issue seemed to connectivity between smoothwall and our two domain controllers. Getting pings every once in a while.

 

Anyway, I've been in since 5am and at 8 I thought I need a break and a coffee. Sat down. 5 minutes later, everything is back up!

 

Well the DCs now talk to smoothwall, smoothwalls fine! As if nothings happened!

 

Although now, two cctv servers and a random fileserver cannot be pinged. They live in separate rooms too.

 

Like someone said previously, maybe power cut and switch config? Its such a strange thing to be happening.

[Davit2005]

Just seems strange that it would suddenly happen at 12.30.

 

I've not found any devices or anything that would affect this.

 

If I ping the domain controllers from my flat network, replies are fine however if I ping Smoothwall, it gets a ping every 3 or 4 replies. I've not tried from another vlan yet.

 

It just feels like a DNS/certificate type issue.

 

I've had to come home now. I'm going to try and isolate some edge switches tomorrow and run some wireshark tests.

 

As far as I'm aware, spanning tree is enabled throughout, however we never touch switch config or mess with the server room so I don't know why it would suddenly start acting up.

 

- - - Updated - - -

 

Thanks all by the way. Feeling pretty useless at the moment but I hope to find some answers tomorrow.

 

Midnight or daytime :-) The chances of a loop been anytime someone is physically in the location. I've only seen it the once. It stood out as the network ports on the switch were going mad, it was one of the few places we did not have spanning tree enabled. This is where it helps if you have out of band management i.e. a console port to check logs of switches ;-)

 

We had multiple vlans and it was only the one having issues. Internet and anything network related, found a patch cable in a room plugged from one wall socket to another.

Edited January 19, 2024 by Davit2005

[chaplic]

FWIW I was asked to be an expert witness at a tribunal of someone sacked for causing (what sounds like) the problem you have. I refused, partially because I know enough, but also know enough to know I wouldn’t consider myself an expert (and, frankly could have done exactly what that poor guy did). So yeah I agree with the masses, feels like a lower-level physical/ looping type problem 100% not cert/ dns, thats layer 7 and this is much lower. On a lan it should be very rare lots of pings are being dropped so its either due to dodgy cabling, the network being busy or just something bonkers.

 

Could be a NIC in something going nuts and screaming electrical noise everywhere. Divide, simplify and conquer.

[FN-GM]

Had similar symptoms once. Turned out that a brief powercut over night caused a switch to reboot and revert to a previous config (I swear I had saved it). Anyway some vlan tagging got messed up and traffic was broadcasting across vlans. It took a while for the symptoms to ramp up though.

 

Get a wireshark trace going and see if you can see any suspect traffic.

 

I have a cron job running on my switches that runs every night and does the following.

 

1. Save the running config to prevent the issue you faced.

2. Save a copy of the config to a SFTP server.

 

You’re welcome to a copy of that config if you like.

[ITGuyNW]

FWIW I was asked to be an expert witness at a tribunal of someone sacked for causing (what sounds like) the problem you have. I refused, partially because I know enough, but also know enough to know I wouldn’t consider myself an expert (and, frankly could have done exactly what that poor guy did). So yeah I agree with the masses, feels like a lower-level physical/ looping type problem 100% not cert/ dns, thats layer 7 and this is much lower. On a lan it should be very rare lots of pings are being dropped so its either due to dodgy cabling, the network being busy or just something bonkers.

 

Could be a NIC in something going nuts and screaming electrical noise everywhere. Divide, simplify and conquer.

 

Yeah this next week will involve trying to figure out what it was.

[paulellam]

Happened to me when an AP went bonkers, happened during the day etc. I did as someone else suggested worked from the core outwards. Disconnected all links to remote cabs, and slowly added them back in, once it went bonkers again, disconnected that link, the went to the cab where the issue seemed to be, un patched everything, plugged in one by one till the system started to play up again and bingo. The AP was an older one, so just replaced it.

[Sephiroth]

We have Cambium wireless and used to have a Smoothwall UTM. The Smoothwall UTM was the cause of most of the unapproved downtime on our network. It got to the point that we couldn't run reports on it or it would crawl to a stop and it was scheduled to reboot once a week to keep it running properly. If you have a Smoothwall UTM, I would be scrutinising what that was doing at the time it all went wrong. Having said that, I wouldn't rule out what others have said about a broadcast storm, caused either by a loop that someone undid afterwards, or with some faulty network gear/cable.

[Chuckster]

Once you figure it out - and I also suspect it to be a loop, disable every port on switches that are not being used. Any data points not in use - unpatch from the patch panel.

 

Additionally, add any unused ports on your switch to a VLAN as untagged with no IP address-helper assigned.

[ITGuyNW]

Examined Smoothwall and nothing in the logs suggested anything. Smoothwall and my two domain controllers couldn't ping each other properly for some reason.

 

If it happens again, I'll have to do what @paulellam suggested and start from the core and do some process of elimination.