+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 26
Wired Networks Thread, Network Issue - What to do now? in Technical; Hi All, Having done a search and reading through various threads I was wondering if anyone could add anymore advice ...
  1. #1

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    Network Issue - What to do now?

    Hi All,
    Having done a search and reading through various threads I was wondering if anyone could add anymore advice to help me track down my problem.

    Around 10am today for no apparent reason our network decided to go down, by down I mean the following behavior;

    -Clients unable to access network resources
    -Clients unable to browse to internet
    -Switches showing a lot of activity (mixture of entry level procurves and a few random unmanged)
    -Unable to ping from one static machine to another
    -Fibre convertors showing lots of activity

    Now at first I thought this must be either a loopback or a virus as Ive seen both have similar affects on a network, Ive taken each server/host (we are virtualized on vmware) and rebooted, also taken all switches in building down. Thought it might our Ruckus wireless controller however this is now back up and appears ok.

    We are back up and stable (ish) now...no idea why or how and Im in the process of troubleshooting further.

    Sophos is showing no viruses
    arp -a resolves as expected
    I can now ping internal resources with no packet loss eg; server to workstation, workstation to server etc
    I can traceroute to google.com successfully

    What I am noticing is that pinging our default gateway (a LA Cisco 2950) I received constant time outs on, this is from any area of the network...would it be reasonable to assume this could cause the issue given nothing has to our knowledge changed on the network?

    Ill be doing another walk around later today to ensure there are no visable loopbacks/damaged cabling etc any other advice or ideas much appreciated

    Thanks for reading

    M

  2. #2

    Join Date
    Dec 2007
    Location
    cumbria
    Posts
    182
    Thank Post
    7
    Thanked 43 Times in 39 Posts
    Rep Power
    25
    The gateway may have blocked you out due to seeing too much garbage,
    probably worth getting the LA to health-check +/- reset from their end.

  3. #3

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Thanks for the reply!

    I have indeed logged a call on the Capita helpdesk as I know a couple of other schools within my LA had their 2950s replaced.

    Hopefully have someone come back to me later today

    Cheers

    M

  4. #4

    SYNACK's Avatar
    Join Date
    Oct 2007
    Posts
    11,038
    Thank Post
    852
    Thanked 2,664 Times in 2,261 Posts
    Blog Entries
    9
    Rep Power
    767
    Could be a broadcast storm thanks to a loopback, try checking for switches patched back into themselves.

  5. #5

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Good call, Ive been obsessed with checking the classrooms, all cabs are under lock and key so nothing should have changed but always worth a look!

    Some great threads about this sort of stuff already , strange thing everything seems stable other than my continual ping to the default gateway.....scary as other than take down individual cabs havent done anything else

    Thanks for all the input thus far guys

  6. #6

    Michael's Avatar
    Join Date
    Dec 2005
    Location
    Birmingham
    Posts
    9,262
    Thank Post
    242
    Thanked 1,568 Times in 1,250 Posts
    Rep Power
    340
    I presume if you disconnect or switch off the Cisco 2950 that problems go away? Other than not being able to ping it of course.

  7. #7

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Hi Michael,
    When disconnecting it earlier it seemed to resolve the issue, although by no means conclusive. I will restarting again later.

    The switch is currently back up and running but as it is managed by the LA I am waiting for them to come back to me.

    Just installing the trial for procurve manager which will hopefully allow me to look at some of the edge switches in more detail, wireshark hasnt yielded anything yet but then Im a first time user so reading up on its usage to ensure Im not making a newbie mistake!

    Thanks!!

    M

  8. #8
    Jamo's Avatar
    Join Date
    Jan 2009
    Posts
    1,349
    Thank Post
    66
    Thanked 175 Times in 147 Posts
    Rep Power
    60
    Any logging on the Cisco or edge switches?

    show logging

  9. #9
    ricki's Avatar
    Join Date
    Jul 2005
    Location
    uk
    Posts
    1,475
    Thank Post
    20
    Thanked 164 Times in 157 Posts
    Rep Power
    52
    Hi

    What I would do is this?

    Do all your servers plug into a main switch?

    If they do disconnect all connections expect the servers into that switch expect the server and reboot the switch. This will see if its the main switch or a server. Check to see if the servers can connect.

    Now connect the switch back up and disconnect all the network feeds to other parts of the school and check that the other switches near the server switch still work. You may have to reboot them. Look at the flashing lights do they go from a blink to a twinkle.

    Check each other cabinet in school to see if they are flashing or twinkling. If they are tinkling connect that arm of the network and check its still working.

    My guess is one arm has a loop. Its the end of the year and kids and teachers are leaving and stopping the network working is fun for some people.

    Once you have it down to one cabinet disconnect all the switches and then connect them one at a time till you get the flash back.

    Dont forget to enable tree spanning or fast tree spanning if they are newer switches.

    Richard

  10. #10
    ricki's Avatar
    Join Date
    Jul 2005
    Location
    uk
    Posts
    1,475
    Thank Post
    20
    Thanked 164 Times in 157 Posts
    Rep Power
    52
    Once this is done

    Capture packets from the network with wireshark for about 15 mins and then anaylse with the free capsa. This will tell you if you are getting broadcast storms duplicate ip addresses network problems.

    You can also get a app for some mobiles that pretend to be the gateway and that will kill your network so it might be worth turning off the wifi while doing tests to eliminate it as the cause,

    You might also want to check that the dhcp server is still giving out addresses as the service might have stopped for some reason.

    Richard

    Richard

  11. #11
    ricki's Avatar
    Join Date
    Jul 2005
    Location
    uk
    Posts
    1,475
    Thank Post
    20
    Thanked 164 Times in 157 Posts
    Rep Power
    52
    Lastly

    Are you having problems with your electrics and do you have ups on the switches as well as the servers as this can cause the switches to become confused a reboot may fix this.

    Richard

  12. #12

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Thanks for all the advice guys overwhelmed by the response

    Ricki my infrastructure runs on 3 hosts (VMware), Ive taken each of the servers down individually as well as restarting the hosts.

    The core switch comes in at a central location with all other switches connected via cat 5 or fibre convertors, have restarted all of them.

    Ill wait till schools out and start investigating further, some great advice here

    S

  13. #13

    twin--turbo's Avatar
    Join Date
    Jun 2012
    Location
    Carlisle
    Posts
    2,334
    Thank Post
    1
    Thanked 381 Times in 340 Posts
    Rep Power
    150
    what's behind the 2950 as this is a pure layer 2 device and not a router.

    Rob

  14. #14

    Join Date
    Jul 2012
    Location
    Swindon
    Posts
    33
    Thank Post
    11
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Hi Rob,
    The 2950 is connected directly to a linksys switch (unmanged and being replaced during summer break), other than that our edge switches consist of some newer procurve models and then its a bit mix and match (netgear, linksys etc).

    Ive just got back from checking each cabinet, definetely no loopbacks

    Checked all classrooms and any other points I know of and unable to find any sign of a loopback on the network

    There is still a lot of activity showing on the switches around the building, Ive turned logging on and am in the process of setting up procurve manager.

    As it stands at the moment the network appears stable, Ive been through the logs on the hosts and there is/was no excessive amount of traffic from the nics/hosts/servers that I can see.

    Is there any other tools I can consider using? Ive left a contiual ping running with timestamp from my iMac to the core switch, The LA have informed me someone will be onsite at some point tomorrow to check out the 2950

    Thanks

    M

  15. #15

    twin--turbo's Avatar
    Join Date
    Jun 2012
    Location
    Carlisle
    Posts
    2,334
    Thank Post
    1
    Thanked 381 Times in 340 Posts
    Rep Power
    150
    Sorry, I mean whats on the outside of the 2950 as this is a layer 2 switch with no routing functions.

    Rob

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 31
    Last Post: 30th June 2009, 04:25 PM
  2. What to do with your old crt's
    By drjturner in forum General Chat
    Replies: 10
    Last Post: 9th August 2007, 05:41 PM
  3. Replies: 3
    Last Post: 24th January 2007, 11:11 PM
  4. Workload. What it used to be and what we do now.
    By Dos_Box in forum General Chat
    Replies: 19
    Last Post: 24th November 2005, 09:41 AM
  5. failed redundancy - what to do?
    By browolf in forum Hardware
    Replies: 3
    Last Post: 2nd November 2005, 08:59 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •