Why would an entire building have 169 IPs but rest of the site is fine?

[TomHD]

Hello All,

 

First major problem at my new school!

 

I have an entire building that is not getting IPs. The rest of the site is running fine! What options do I have to troubleshoot?

 

I have reset the switches across the site and restarted the DNS and DHCP services with no luck. I have The Dude but not sure what I am looking for. Some help with this will be very much appreciated. Unfortunately nothing is labelled and I cannot see a MAC address on the offending switches!

 

Regards, TomHD

[localzuk]

That sounds like the link between the building and wherever it connects to isn't working.

[fairm010]

That or you have a faulty switch in there.

[TomHD]

I have 2 switches in 1 comms box, both show power and data transmission! No sign of rogue DHCP in the DHCP logs

[SHimmer45]

169.xxx.xxx.xxx are the clients self assigning IP if they are not getting a response from your DHCP server or there are no longer any addresses available in the scope.

try giving a machine in the area a static address and then attempt to ping your DNS/DHCP/AD servers if you get a response i would be looking at the DHCP scope.

 

check links between blocks back to your "sever room"

check for any loopbacks in the building with the problem.

[tmcd35]

Is DHCP full? are they on the same VLAN (if used) as the DHCP server? Are you 100% certain, as in cable tested, the link(s) between effected edge switch and core switch are not faulty? If you have spanning tree enabled and the lights on the effected switch are going crazy have a look and see if you have a network loop or a faulty NIC in that building.

[Tesla]

@tmcd35 beat me to it, but +1 for network loop possibility

[chazzy2501]

is it possible that the switches are preventing a broadcast storm and have killed multicast?

 

get wireshark running over there.

[TomHD]

Ok, so some machines now have a 10.xxx.xxx.xxx IP but have no internet access although windows is saying there is! I've plugged a Mac Book directly into the switches in the offending building and its gets a 10.xxx.xxx.xxx IP but no Internet again.

 

I have tried assigning static IP to a machine in the building and pinged the DHCP but host was unreachable.

 

Thanks for the replies so far

Edited September 5, 2014 by TomHD

[Dos_Box]

What type of link do you have between the building and rest of the site? Also, what make\model of connection kit do you have?

[TomHD]

I am unsure of the links between. Unfortunately I have only just taken over and do not have enough contracted hours to quantify what is in place. As said... The Muppet who installed labelled nothing!

 

Now installing WireShark on the MBP and will run attached to the switches in the affected building via Ethernet?

[Dos_Box]

Do you even have access to the switches (passwords etc.)?

[TomHD]

I have access to the switches that have the MAC address labels eg the Netgear switches in my server room and can access them. The switches in the offending building do not show their MAC so cannot determine if they are seen by the network using netscan.exe

[Tesla]

set up spiceworks and run an inventory scan? it can detect switches etc.

[caffrey]

Netgear ? It really is netgear self destruct week after all!

 

Smartwizard discovery might help

[TomHD]

The Netgear switches are fine. The switches in the offending building are Planet FGSW 2402-rs. The lights are going crazy and loading windows takes an age.... Broadcast storm?

[jamesreedersmith]

Sounds like a broadcast storm - unplug all cables, reboot switches and gradually repatch until issue reappears, then resolve issue.

[north-ict]

Ok, so some machines now have a 10.xxx.xxx.xxx IP but have no internet access although windows is saying there is! I've plugged a Mac Book directly into the switches in the offending building and its gets a 10.xxx.xxx.xxx IP but no Internet again.

 

I have tried assigning static IP to a machine in the building and pinged the DHCP but host was unreachable.

 

Thanks for the replies so far

 

Is that due to proxy settings not set up, [if you are using a proxy on site ???] (Ignore me if I am way off)

[TomHD]

Just to clarify, on the Planet switches in the offending building I have solid amber lights and frantically flashing, in unison green lights. Broadcast storm?

 

I will be coming in this weekend to carry out jamesreedersmith solution

[localzuk]

That sounds just like a broadcast storm, yeah.

[tmcd35]

The Netgear switches are fine. The switches in the offending building are Planet FGSW 2402-rs. The lights are going crazy and loading windows takes an age.... Broadcast storm?

 

99.99999999999995% certain of a broadcast storm. Most likely culprit is a student plugging a network cable between two network points - grrrr. Next most likely is a faulty NIC sending out bad packets. Easiest way to find out were (assuming points are labelled and correctly) is to unplug cables until the lights calm down. The last one unplugged before the lights stopped going crazy is your culprit!

 

Also, thank the Edu Gods that you have spanning tree enabled otherwise your entire network would have been down!

[Dos_Box]

Planet FGSW 2402-rs? Seriously? Someone put in budget 10/100 switches?

It may even be a capacity issue. I'd definately look at swapping out one of those to test whether it may be a bandwidth problem or simply rubbish hardware.

 

Edited September 5, 2014 by Dos_Box

[TomHD]

Yeah, the rest of the hardware throughout is good stuff. Thanks for letting me know Planet are budget crap, I'll get and replace them next budget

 

There is no labelling so that will be my first job in the morning. I will then unplug, power down, power up and plug 1 by 1.

 

Funnily the caretaker knew exactly what I was talking about as he has heard of it happening in other schools!

 

Thanks for all your help guys, I will be back with good or bad news tomorrow

[TomHD]

So some further investigation has revealed that the offending building is served by a 48F SFP Link in the server room (on a Netgear GS748T) to another GS748T. This other GS748T then serves the Planet switches upstairs. If I remove the link between the Planet switches and the GS748T downstairs the storm stops on the Planet switches, it continues on the GS748T downstairs. If I then remove the link to the GS748T downstairs from the GS748T in the server room the storm stops on the one downstairs.

 

Does this mean that the offender is connected to the GS748T switch in the server room?

 

Many Thanks, Tom.

[Dos_Box]

I would try disconnecting all ethernet cables from the remaining switch in the server room one by one to see if the storm orginates from there (I take it the switch in the other building it links to is fine?). If you are still getting a storm in that switch, but not the one at the other end then try doing a hrd reset i.e. disconnect from mains and repower. If the storm still remains try swapping over the GBIC modules to check that you don't have a faulty one.

If you have access to teh switch WEBMIN you can check which port is generating the most traffic.

Also try adjusting the GBIC duplex options if they are available (if the problem is there of course).

Just to suggest a few options of course.

At least you are narrowing this down and I hope you get to the bottom of the problem today