newpersn Posted June 17, 2014 Posted June 17, 2014 I don't know how to word this Every now and then we get a whole site network dropout. It only last for a few seconds but enough for the IP phones to restart. (Takes around a minute to start back up) We have vlans and I have walked round the site and done a quick look back checks and not seen anything. Where do we start? Kinda strange. If it was a switch it would only effect that vlan. (Unless it's the main switch) Help
jamesreedersmith Posted June 17, 2014 Posted June 17, 2014 is it a power blip affecting the cabinet and therefore the switches restarting - are they on a UPS?
newpersn Posted June 17, 2014 Author Posted June 17, 2014 is it a power blip affecting the cabinet and therefore the switches restarting - are they on a UPS? We dont have any UPS. But if it was a network switch restarting it would take longer than a minute to restart. Computers reconnect within seconds of it happening.
jamesreedersmith Posted June 17, 2014 Posted June 17, 2014 Depends on the switches - some unmanaged/older switches will restart almost instantly.
newpersn Posted June 17, 2014 Author Posted June 17, 2014 Depends on the switches - some unmanaged/older switches will restart almost instantly. None of our core switchs are unmanaged and there a only a few years old (3com/HP) The switchs that are connected to our VLAN are HP and are only 2/3 Years only
FN-GM Posted June 17, 2014 Posted June 17, 2014 The phones shouldn't power off, are they POE? Even if network connectivity is lost and the switch is still powered up the phones should remain powered. Is this all on 1 switch or 1 stack?
jamesreedersmith Posted June 17, 2014 Posted June 17, 2014 The phones shouldn't power off, are they POE? Even if network connectivity is lost and the switch is still powered up the phones should remain powered. This is why i think its a switch/es rebooting, seen it before. can you put them on a different/UPS'd supply for a while to see, or set a continuous ping to the switch and see what happens to this, can you enable a syslog/event log on the switch.
newpersn Posted June 17, 2014 Author Posted June 17, 2014 The phones shouldn't power off, are they POE? Even if network connectivity is lost and the switch is still powered up the phones should remain powered. Is this all on 1 switch or 1 stack? We have mix analog and poe phones. The POE phones display "DHCPSVR NOT FOUND" My deskphone is a analong phone which isnt connected to the network. Just runs straight to the phones. i cant say its 1 switch or stack as the phone are all over the site. (diffenent building connected by fibre)
chinesewhispers Posted June 17, 2014 Posted June 17, 2014 We have this, where a particular cab with 2 or 3 switches loses power, this behaviour occurs when it powers back on. The cab contains a switch with trunked fibre connections to the core switch, and also the first link to the VOIP server. Not sure whether it's the order the switches boot up in (time taken), or something else, but it was the second time it happened that I managed to pin it down. Electricians were doing some work and killed the power to it both times. Really does seem to generate full network traffic bursts/outages for a minute or so until it settles. How often does it happen?
newpersn Posted June 17, 2014 Author Posted June 17, 2014 This is why i think its a switch/es rebooting, seen it before. can you put them on a different/UPS'd supply for a while to see, or set a continuous ping to the switch and see what happens to this, can you enable a syslog/event log on the switch. The switchs are connected to the same supply that powers the servers. I would of thought it would trigger the servers to trip out if its a spike.
FN-GM Posted June 17, 2014 Posted June 17, 2014 It could be the phone system or type of phones causing them to power off when they loose connection. Have you gone to have a look to see if you backbone has bandwidth? As they are only 3 year old I assume its a 10GB backbone? I managed to do similar (without the phones) in a test environment. SCCM was hogging the bandwidth.
Dantech Posted June 17, 2014 Posted June 17, 2014 If I were you I would buy a cheap APC ups for EVERY switch cabinet in the school. I did this in the school I worked in and it made a huge difference. We often had power cuts which were brief and this had no effect on the switches. A UPS will provide a constant reliable power source. A layer two edge switch (HP) uses very little power so a cheap £50 APC UPS will be more than adequate - just do the maths and work out the maximum power consumption and ensure the UPS can deliver more than needed. You will have to replace the batteries every three years but it will save a lot of problems.
newpersn Posted June 17, 2014 Author Posted June 17, 2014 It could be the phone system or type of phones causing them to power off when they loose connection. Have you gone to have a look to see if you backbone has bandwidth? As they are only 3 year old I assume its a 10GB backbone? I managed to do similar (without the phones) in a test environment. SCCM was hogging the bandwidth. Ok. We don't have a 10gb backbone. Just 1gb. But we never have any problems before now. Nothing changed. I'm playing with sccm but that's only on a hyper-v on my pc in the office. Moving away from the phone that's just part of the problem. Myself and the nm will start checking the logs on the switches. The dhcp/dns/ad are not showing any problems in the logs. I think it's a switch on its last leg.
FN-GM Posted June 17, 2014 Posted June 17, 2014 You won't get any logs from DHCP DNS or AD to be honest. 1
newpersn Posted June 17, 2014 Author Posted June 17, 2014 It happened around 10 timed today. Started Around 845 this morning with few staff logged in.
Dantech Posted June 17, 2014 Posted June 17, 2014 Can you not just web manage into the switch and check the traffic on each port - none of then should max out? Run a wireshark on the network segment and see if anything strange is going on.
IT_Ninja Posted June 18, 2014 Posted June 18, 2014 Even if there's a spike in power it is likely to cause some form of disruption. I would recommend securing a UPS since it compensates for spikes, outages etc - it is quite integral to keeping things going. If you have any old UPS units you can get them going by buying the batteries yourself and saving a few bob, otherwise a standard UPS is worth it's value in the long term. (I've even got one for my PC)
newpersn Posted June 18, 2014 Author Posted June 18, 2014 I think we had found a problem with a switch. We have quickly swapped it out and replaced it and it all seems fine so far today...
IT_Ninja Posted June 18, 2014 Posted June 18, 2014 I think we had found a problem with a switch. We have quickly swapped it out and replaced it and it all seems fine so far today... Oh yeah... it slipped me... my NM mentioned having a similar problem and it was one of the switches. Funny thing it didn't give any signs of failure but was the root cause. Replaced and problem solved. I'll ask if any symptoms for sake of curiosity. 1
newpersn Posted June 18, 2014 Author Posted June 18, 2014 Oh yeah... it slipped me... my NM mentioned having a similar problem and it was one of the switches. Funny thing it didn't give any signs of failure but was the root cause. Replaced and problem solved. I'll ask if any symptoms for sake of curiosity. We this switch does a vlan over fibre. But we did get one call this afternoon saying the phones went down but I can't trace that problem. (The guy on our reception isn't the brightness spark in the world)
newpersn Posted November 20, 2014 Author Posted November 20, 2014 (edited) Hate to bring it up.... But its Back...... And its seems to be constent aswell.. Just looking though logs of switches now Edited November 20, 2014 by newpersn
m25man Posted November 20, 2014 Posted November 20, 2014 Watch out for a failing Fibre Module! I had this recently with an Intermittent LAN drop out fortunately it was happening at the edge of the network and only affected a few devices that were not in use regularly. We spotted it when the Unifi Server started to report an AP disappearing regularly but the AP of course remained UP and radiating! Turned out to be a compatible SFP module and when removed you could smell it had burned out! I have also recently had an intermittent link drop elsewhere, which turned out to be the local vermin had taken a liking for a section of fibre cabling! Eventually the link died and in replacing the affected fibre drop discovered the well chewed section of fibre of which 2 cores were still intact. So it needn't be an overly complicated networking issue, sometimes the physical ones can be as hard to diagnose.
cscc Posted November 20, 2014 Posted November 20, 2014 You mention in your original post this is effecting all devices on site? The main culprit here sounds like your core switch(es) are at fault if everything is losing connectivity.
cscc Posted November 20, 2014 Posted November 20, 2014 You could setup smokeping to monitor all switches on a Linux box to monitor drops/packet loss to potentially identify the cause. SmokePing - About SmokePing
newpersn Posted November 20, 2014 Author Posted November 20, 2014 You mention in your original post this is effecting all devices on site? The main culprit here sounds like your core switch(es) are at fault if everything is losing connectivity. Looking the core switches. We disable the port its coming in from and it find another port to flood. This was taken from an middle switch Instance0'sportGigabitEthernet1/0/49wasnotifiedofatopologychange. Tones of those messages
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now