Network failure

[chrbb]

Everything gone smoothly for first two days back, then mid morning today network seems to go down. I was installing software on a laptop from the domain server software drive when I lost network connection, went to check the server, it was still up and I could log on, internet access still available but I couldn't ping the wsus server or access any of the computers on the network. The wsus server was still up but couldn't ping the domain server.

 

Suspecting a loopback I walked round school, couldn't see anything, no computers in school had access to either of the servers. Rebooting the servers didn't make any difference, so I went back round school turning off access points, laptops and computers (very unpopular at this point), then unplugged patch cables from the switches of the computers that weren't turned on yesterday (if that makes sense). At which point the two servers could ping each other.

 

I then turned everything back on indivdually and replaced each patch cable in the switches, all is now ok but I'm non the wiser why the whole network failed.

 

Any ideas what I should be looking for to check it doesn't happen again.

[twin--turbo]

Were any of the switches showing broadcast storms?

 

Do you have STP? and do you have BPDU guarding?

 

It's possible somone created a loop, filled the network with a broadcast storm and then removed the link.

 

Do your switches have history loggs?

 

Rob

[chrbb]

No sign of broadcast storms, they are bog standard unmanaged switches.

[twin--turbo]

Well next time it happens run round every switch and see if one is going balistic...

 

If that shows no obvious sign of issue then I would be disconnecting everything from the switch your servers connect to appar from the servers, if pings are not restored then you either have a faulty switch or a conflict.

 

Rob

[chrbb]

Well next time it happens run round every switch and see if one is going balistic...

 

If that shows no obvious sign of issue then I would be disconnecting everything from the switch your servers connect to appar from the servers, if pings are not restored then you either have a faulty switch or a conflict.

 

Rob

 

Thats basically what I did, only tbings running were the servers, at which point they could ping each other, then turned everything back on in small groups.

[cpjitservices]

Have you got Spanning-Tree turned on ? What switches have you got ?

[twin--turbo]

NO STP....

 

they are bog standard unmanaged switches.

 

 

 

Rob

[cpjitservices]

Ohhh aye sorry - must of read that post too quick!

[twin--turbo]

If you can find a small budget to at least upgrade to a smart switch for the servers that will start to show some diagnostic benefit. Good second hand switches are on ebay all the time too , you can get a lot of specification for a little bit of money and if it saves half a days lost teaching then it is money well spent.

 

Rob

[SYNACK]

Could be a switch on its way out, we had that on Tuesday. A fully populated 48 port switch decided to off itself in the morning, unfortunately every big high volume printer in the place was on it, eep, cue enraged people very nearly smearing their own blood on paper just so they could consume their daily dosage of tree.

 

You can sometimes bring back a crashing switch by yanking the power for 10 or so seconds then plugging it back in.

[zag]

As others have said, buy a few managed network switches, turn on STP option and you will never have loopback problems again

 

Otherwise, in this case again, start a ping -t to the server then go round and isolate each switch (or port). As soon as you find the broken one the ping should restart.

[twin--turbo]

Our first switch we had made by planet (going back probably 10 years), Randomly started locking up it's uplinks to our remote buildings..

 

And about 2 years ago we had an 18 Month old Cisco 3560-24POE kept randomly locking up, a fair amount of time was spent on that including an IOS update and eventualy managing to rais a TAC with cisco via the reseller and get it replaced. It's little known but cisco fixed configuration switches do come with a limited lifetime warranty.

 

Rob

[SYNACK]

It's little known but cisco fixed configuration switches do come with a limited lifetime warranty.

 

Rob

 

And no free firmware updates, getting Cisco is expensive, keeping it is even worse. Their gear is usually epic though.

[Destinova]

And no free firmware updates

 

Incorrect. All Cisco Desktop Switching Business Unit switches (2K, 3K) come with IOS upgrades at no additional cost, for the lifetime of the product. This includes both fixed and stackable versions.

[SYNACK]

Incorrect. All Cisco Desktop Switching Business Unit switches (2K, 3K) come with IOS upgrades at no additional cost, for the lifetime of the product. This includes both fixed and stackable versions.

 

Well they have changed that then, where do you sign up for that and is it avalible outside the US?

[twin--turbo]

Download Software - Cisco Systems

 

Been able to get some IOS's for at least 2 years.

 

Rob

[twin--turbo]

Router software is still restricted download though.

 

Rob

[localzuk]

Incorrect. All Cisco Desktop Switching Business Unit switches (2K, 3K) come with IOS upgrades at no additional cost, for the lifetime of the product. This includes both fixed and stackable versions.

 

I've *never* heard this. I have been told that you have to have a support contract to get the upgrades though, by many different sellers and users of Cisco gear over the last 15 years...

[dwhyte85]

A badly damaged cable can do some meaty damage, as can a damaged switch - do you have a big estate or can you quickly isolate a bad device/cable?

 

RE: Cisco upgrades: if you register the product online against your Cisco account, you should be eligible for upgrades? I know I was for ASA 5510s.

 

Best of luck.

[Destinova]

I've *never* heard this. I have been told that you have to have a support contract to get the upgrades though, by many different sellers and users of Cisco gear over the last 15 years...

 

No contract or product registration required for current generation 2K and 3K switches, just a valid CCO login. Check the data sheets, etc. for your particular model. As an example:

 

2960 and 2960-S Series

 

Q. Can I obtain a Cisco IOS Software updates for the Cisco Catalyst 2960 Series at no additional cost?

A. Yes. Cisco offers ongoing Cisco IOS Software updates for certain fixed-configuration and stackable Cisco Catalyst switches at no additional cost. For the life of the product, updates within the Cisco IOS Software package purchased (LAN Lite and LAN Base) will be made available.

Q. How do I get an update for the Cisco IOS Software for the Cisco Catalyst 2960 Series at no additional cost?

A. Visit Cisco Systems, Inc, click "Downloads," and select "Switch Software." Downloading software requires a Cisco.com username and password. If you do not have a Cisco.com username, you can obtain one by clicking "Register" at the top of any page on Cisco.com.

 

Edit: This was Cisco's response to HP's "lifetime warranty" on ProCurve switches. Limited Lifetime Warranty and free software. The LLW just means you have to be the original owner, and it doesn't cover the power supply past 5 years. Otherwise, they'll RMA/cross-ship a new unit *without* any support contract with 5 business day coverage.

 

It's certainly not the way a lot of people are used to Cisco software, but it's been around for a couple of years. I know I first heard of it at Cisco Live (US) in 2010.

 

Obviously this doesn't cover your Cat6K's or Nexus gear but access-layer hardware is covered.

Edited September 21, 2012 by Destinova