Help!!!

[ndavies]

OK, major nightmare happening.

 

Half the machines on the cc3 network can't see the dc's. Ping t tests fall over after exactly ten minutes, removing the lead and reinserting sets it going again. Across the entire site, I have machines in the same room, some which can ping and see the dc's, somewhich can't. So I don't think it's a local switch problem.

 

We've tried to eliminate loopback by visiting each and every point and cable in the whole school, to no avail.

 

One machine connected directly to the core, which the dc's are on, ping to the dc's fail after ten minutes

 

A second machine connected to a switch which is connected to the core, ping to the dc's fail after ten minutes

 

A third machine, at the end of a fibre link across the site, does exactly the same.

 

We've double checked dns, no issues, got RM to check the dc's over, no problems.

 

Has to be a core switch falling over or an unfound lookback, right?

 

I'm running out of things to look at now, Any advice greatly received!

 

Add this to the fact that a certain telecoms company trashed our ATM link for our ADSL for a week, and I'm lovin' life at the minute.....

 

Nick

[pete]

Unplug all other switches from the core switch. Reboot core switch. Test pinging across core switch. Works or not?

 

If yes, plug another switch back into the core and test.

If not, look at cpu use on the core switch. Look at packet counts / bad traffic per port and investigate. Also check the fans are running / temperature is ok.

 

Rinse, lather, repeat until you find the affected switch, which should aid loop discovery.

[waldronm2000]

Might be worth checking ARP and DNS caches on a PC that can ping and one that can't, as the 10 minute thing sounds suspiciously like a cache timeout.

[SYNACK]

Could be broadcast storms caused by a loopback, use wireshark to check on that one.

 

What about the DCs and other servers, can they stay in contact with each other for more than ten minutes. Could be a broadcast storm isolation feature on the switches misbehaving being linked to the STP settings of the switch as obviously the servers would be setup on the switch differently.

 

The ten minute window is strange though, do you have any active AV running on the servers that could be isolating the stations after a set period of time, the fact that an unplug/replug fixes it is strange though.

 

Can you plug just the servers and a single workstation into a switch and test connectivity then to rule out client or server side issues.

 

The other thing that you could try depending on your switches is log on to one of the client switches managment interfaces and enable logging so that you can see if the switch itself is doing something to the connection at the ten minute window.

 

It does certainly sound like a strange issue though.

[ndavies]

Unplug all other switches from the core switch. Reboot core switch. Test pinging across core switch. Works or not?

 

If yes, plug another switch back into the core and test.

 

 

Spent nearly nine hours doing this yesterday,

 

got the core with servers and one ws, no problem. got the second switch with servers on it and one ws, no problem,

had the core with the servers on it, the second switch connected with one workstation, and the third switch into the core. workstation Fell over. (it was twenty minute periods yesterday, today it's ten)

Took half the patches out of third switch, fell over

Took the remaining patches out of the third switch, fell over

Took the third switch out of the core, still fell over!

 

Took out the third switch. Still fell over!

 

If not, look at cpu use on the core switch. Look at packet counts / bad traffic per port and investigate. Also check the fans are running / temperature is ok.

 

temp seems ok, no unexpected/ out of the ordinary noise

I can't get to the core switch config, as I have no idea what the IP is. can't find it in dhcp, spent two hours last night, manually going through every address in dhcp that wasn't already leased with a workstation name. It's a netgear gsm7212. if you say GSM phonetically, it pretty much sums it up.....

[waldronm2000]

You could use an IP ping sweeper to identify all hardware on your subnet, but I still suspect it's not the switches; more likely something like intermittent ARP failure.

[SYNACK]

It may not have even been configured and could still be on the default address whatever that is. I did a quick look on the netgear site but they don't publish a manual for it on there.

 

Does it have a console port on it so that you can logon to it locally with a serial cable, that might be quicker than working out the ip.

[ndavies]

It does have the console port on it, but I would have no clue how to go about connecting etc. Any advice on this greatfully received.

 

Basically, tonights efforts have resulted in..

 

We've setup the servers, second and third switches and test workstations on a spare switch which has proved to be stable, unfortunately this has no fibre ports so I can't rule out the possibility that whatever is causing the problem coming from the 5 fibre links we have to the rest of the site. But at least it's down to the fibre or the core switch. Definitely not the second or third switches in the server room cab or something on them.

 

Tomorrow I'm going to try to rule out individual fibres out one at a time adding them to the spare switch, using a single media converter. If each prove to be stable, then it's either the core switch or a combination of the fibres.

 

Will investigate the packet software etc. Did try Wireshark on me Mac, but it couldn't find the nic and I didn't feel like spending hours to sort that out.

 

Cheers!

 

Nick

[pete]

Who's responsible for the core switch? External support contract or internal? Find out what firmware revision it's on. latest firmware

 

The GSM7*** series had some really craptastic firmware revisions in the past which would produce behaviour similar to what waldron2000 is suggesting. One of ours did it out of the blue a few years ago.

[ndavies]

Who's responsible for the core switch? External support contract or internal? Find out what firmware revision it's on. latest firmware

Er...that would be me. This switch apparently pre-dates at least two of my predecessors in this job. There's no support contract.

 

 

The GSM7*** series had some really craptastic firmware revisions in the past which would produce behaviour similar to what waldron2000 is suggesting. One of ours did it out of the blue a few years ago.

 

Will take a look see. Thanks

 

any suggestions for getting it's IP discovered. I've gone through literally every address in dhcp leases, so it must be configured with a static or nowt. Just reading up about cli commands...ah well, all good for the CV!

 

Nick

Edited June 3, 2010 by ndavies

[IanT]

Angry IP

[p858snake]

it may not be in the same scope as what your using, depending on how your ip schema is set up.

 

Try something like 192.168.X.{1 or .254} or 10.0.X.{1 or 254}, First and last useable hosts generally seem to be the best for these things.

[srochford]

Do you have a serial cable to connect to the switch? If not they're cheap and you do need one - Videk Online

 

You probably need the part 4070 on that page (9 pin female to 9 pin female) but if your laptop doesn't have a serial port then you also need a USB to serial converter - Videk Online

 

If you're running XP then Hyperterminal is already available (may need to add it in Control Panel) or you can just download a free edition - Free HyperTerminal Private Edition Download - you'll need this for Windows Vista or Windows 7

 

If you've only got a Mac laptop then things are harder because I don't think you'll have a serial port and the USB to Serial adapters don't always come with Mac drivers.

 

Connect the serial port to the switch, set the baud rate in your terminal program to 9600 with 1 stop bit and no parity (it's just drop down menus; not as scary as it looks)

 

When you've done that, just press Enter a couple of times and the switch should respond. If it's asking for a password then you're probably stuck if no-one has kept a record of what they set (the default is username admin and no password so give it a try)

 

You can then get the IP address and browse to it from a web browser.

 

I'd really try and get Wireshark working - it's one of those tools that's essential at times like this. As others have said, it sounds like some kind of packet storm and Wireshark will show you that (you don't have to understand it - basically, you have wireshark running with the core switch and server etc and you'll see little traffic. As you plug in other switches you'll see more traffic and when you plug in the switch leading to the fault you will see mega traffic. That then points you to where the fault is and you can start isolating from there)

 

Good luck

[pete]

If you've only got a Mac laptop then things are harder because I don't think you'll have a serial port and the USB to Serial adapters don't always come with Mac drivers.

 

He really shouldn't need drivers. Something like (from a terminal):

 

screen /dev/ttyUSB0 9600

 

(where ttyUSB0 is the usb-serial cable - will differ on OS X - look in /dev for likely suspects). You'll have to set the speed because (iirc) OS X picks retarded defaults..

[waldronm2000]

You can then get the IP address and browse to it from a web browser.

 

This may be obvious but worth mentioning I guess... If you do manage to find an IP address this way and it's not on your local subnet, then you'll either have to reconfigure it at the CLI to be on your subnet (best option) or reconfigure your laptop's IP to be on the same subnet as the switch and plug straight in before browsing. Oh, and bypass proxy of course.

 

Good luck.

 

One other thought I had... this is almost exactly the behaviour you'd get if there was a hardware device somewhere on your network configured with a static IP the same as your DC, and with a filter blocking ICMP. Only difference is you'd expect the DC to report the duplicate IP.

[ndavies]

Some progress this morning. I've managed to get onto the switch via the console port and get the IP etc.

 

Except it's on an old subnet, we changed the subnet a few years ago to increase our ip range. I manually re-confiugured it to the new subnet, and could connect fine in a web browser on a workstation. Got the password so that's ok.

 

On Hyperterminal it came up with the message arptew failed on a0f5010. Who can tell me what IP this is?! hex conversion!

 

 

On the web config, I found STP was enabled on the switch and every port, but on fibre port 3, the spanning tree stats were off the scale.

 

I reset the switch, and just looked at the stats again. All ports apart from 3 are still at zero, port 3 is increasing.

 

the stats were

 

STP BPDUs transmitted 1156

STP BPDUs received 3

 

 

Presumably it's a loop on this fibre, or the port 3 on the switch is foobared. I note that on port 5, there is activity light on the switch with the others, but the light next to the port is out, and the web config shows that it's out.

[RinSewand]

On Hyperterminal it came up with the message arptew failed on a0f5010. Who can tell me what IP this is?! hex conversion!

.

 

I get 168775696, which may be wrong, but either way I don't think its a valid IP.

 

Only skim read the thread - but is Port3 there for redundancy or is it a link to another switch? Something been fubared in terms of the redundancy config?

[SYNACK]

Could be the switch at the other end of the link could be having issues and responding strangely to STP requests. I would turn off that switch, power cycle the core then power it back up again and see what it does. The address looks like half of a MAC address but it is probably not linked to anything. I would also agree with the firware upgrade idea and unless you have anything like VLANS or trunking setup perhaps wipe the config back to factory default. I know that some stuff ships with horrific bugs in it that are only fixed with later revisions of the software.

 

The high nuber of BPDUs indicates that it is trying to set up STP but does not look like it is suceeding and/or the packets are going missing.

 

If it was a hex converted IP it could would work out to 10.15.80.16

Edited June 4, 2010 by SYNACK

[waldronm2000]

Apparently an arptew failed message (never come across this one before) gives a hex representation of the IP address, so it's definitely 10.15.80.16. There's something vaguely related at EPICS Tornado/vxWorks Information near the bottom of the page, but not sure it will help.

 

Could be STP incompatibility between switches, so trying to resolve a loop but failing.

[RinSewand]

OOooops - worked it out as a full number as opposed to binary... doh!

[ndavies]

Ok, indeed 10.15.80.16 was correct, it was a laptop I was using to run Wireshark and connect to the switch through the console port. Not sure quite why that was failing..but...

 

Set up servers again on spare switch, put machines on, 2nd and 3rd switch on, and only having one spare fibre port to work with, went through each of the fibre connections (going to the rest of the site), left each one on for thirty minutes and the pings and connections didn't drop once,

 

the only one i can't be sure of is a link which appears to have broken this afternoon (it never rains but it pours eh?!) we had it up and running this morning fine, but this afternoon there's no activity on it. The annoying thing is this is the one with the router on, the router that has been off for the past week because of BT's ineptitude...grrrrr...

 

So...in short

99% sure it's the core switch dying, particularly on port 3 (the fibre and module going into that port was absolutely fine).

ordered a new switch for delivery tomorrow, stick it in on sunday with the cabling dude checking out the faulty link to the router's switch and should be cool for monday morning.

 

not out of the woods yet, but thanks one and all for your excellent advice and help in this, the worst week of my professional life!!

 

Cheers for now. I'll update on Sunday. Pints on me if you lot ever swing by South Manchester.

 

*when I said swing, I meant..you know, pass by..not the car-keys-in-a-fruit-bowl and lights out kind of swinging.....

Edited June 4, 2010 by ndavies

[ndavies]

This may be obvious but worth mentioning I guess... If you do manage to find an IP address this way and it's not on your local subnet, then you'll either have to reconfigure it at the CLI to be on your subnet (best option) or reconfigure your laptop's IP to be on the same subnet as the switch and plug straight in before browsing. Oh, and bypass proxy of course.

 

 

You were spot on with this btw, it had the old subnet address, before we changed it to get a bigger ip range.

[teejay]

Ok, indeed 10.15.80.16 was correct, it was a laptop I was using to run Wireshark and connect to the switch through the console port. Not sure quite why that was failing..but...

 

Set up servers again on spare switch, put machines on, 2nd and 3rd switch on, and only having one spare fibre port to work with, went through each of the fibre connections (going to the rest of the site), left each one on for thirty minutes and the pings and connections didn't drop once,

 

the only one i can't be sure of is a link which appears to have broken this afternoon (it never rains but it pours eh?!) we had it up and running this morning fine, but this afternoon there's no activity on it. The annoying thing is this is the one with the router on, the router that has been off for the past week because of BT's ineptitude...grrrrr...

 

So...in short

99% sure it's the core switch dying, particularly on port 3 (the fibre and module going into that port was absolutely fine).

ordered a new switch for delivery tomorrow, stick it in on sunday with the cabling dude checking out the faulty link to the router's switch and should be cool for monday morning.

 

not out of the woods yet, but thanks one and all for your excellent advice and help in this, the worst week of my professional life!!

 

Cheers for now. I'll update on Sunday. Pints on me if you lot ever swing by South Manchester.

 

*when I said swing, I meant..you know, pass by..not the car-keys-in-a-fruit-bowl and lights out kind of swinging.....

 

With fibre it could well be that they've been left on auto-negotiate the link speed and duplex, which causes problems with some switches. Set the speed manually on both switches and you may find it sorts the problem.

[ndavies]

Finally got to the bottom of the problem today.

 

First a history lesson..

Our Internet connection died two weeks ago, the ATM circuit had gone, but it took a week to resolve.

In the meantime, our internet provider sent us a replacement router as a troubleshooting step.

The media converter that we use to link the segment which holds this router, went down on Friday, preventing me from testing the only fibre link I hadn't ruled out, leading me to believe it was the core switch on the blink (which I still believe it is but only on port 3, now inactive on the network)

 

This replacement router was the cause of the ten/twenty minute workstation connectivity issue. How? Why? I don't yet know, but after 6 and a bit hours of repairing the link and fault finding today, we pulled the router out of the network and the issues went (even with the router connected to the network switch, but the ADSL line removed from the router, the issue persisted).

 

Network connectivity now totally restored and stable across the site. Internet speed still slow, but that will be in the hands of our provider first thing tomorrow. I'm claiming this as a victory!

 

Thanks y'all. You've been ace.