Random issue - DHCP clients receive IP address but can't ping anything?

[themightymrp]

I'm pulling my hair out on this one.

 

Over the Christmas break I implemented VLAN's on my network where we had none previously. After various tweaks to settings it was finally working. Clients in each DHCP scope pick up the relevant IP addresses, WiFi clients get the relevant IP dependant on their SSID, even AirGroups is working through the Aruba system so that we can use Airplay on the iPads to AirServer software.

 

However I'm randomly getting the issue mentioned in the thread title. And it can be on any machine in the school so no pattern. The client picks up an IP address just fine but then it doesn't "see" the network. I can't ping anything, including its default gateway on the core switch.

 

I've tried various ideas and they seem to work in the short term, but then the same machine may get it again later.

 

Things tried:

 

1) Manually setting the IP address on the machine

2) Reinstalling the network card

3) Running ipconfig /flushdns

4) Running arp -d *

5) Running nbtstat -R

 

Each of these has been hit and miss and has sometimes made it work and others not. The DHCP lease time is set to the default 8 days with scavenging DNS records the default 7 days (or whatever the defaults are).

 

The 'core' switch is a HP 5130 and is doing all the routing for the VLAN's. It then statically routes anything not in our ranges to the Smoothwall which has routes added back to each IP range (had some help getting that bit to work from Schools ICT).

 

Anybody have any bright ideas or know where I can check things? We have 3 domain controllers - 1 virtual and 2 physical. One of the physicals is the DHCP server.

 

HELP!!!!!

[themightymrp]

Also worth adding, I have run the dcdiag tests on the domain controllers and all tests come back fine - apart from one error in the SystemLog test about being unable to communicate via DCOM to the IP address of the Smoothwall. But looking into that, it is likely expected behaviour as COM+ will be blocked on the firewall.

 

Had another 3 machines with the issue this morning

[themightymrp]

Just had a couple more. On one of them I was actually able to ping the VLAN's default gateway but nothing more - not even other PC's on that VLAN. Could it be that my core switch is a) incorrectly configured or b) can't cope?

 

Manually setting the clients DNS server to just point at the DHCP server seems to help fix the issue quicker, but it is by no means the answer.

 

Anybody any thoughts?

[IrritableTech]

Just to confirm it isn't a DNS issue - you are pinging IP addresses of servers and clients from effected clients rather than names right?

[themightymrp]

Yep tried pinging both IP and names. Just had another couple doing it. There doesn't seem to be any kind of pattern at all and no clear method of resolving it. After a fair bit of repeating what I mentioned in the opening post it starts to work.

 

I'm just going round in circles right now. I just don't know where to check!

[IrritableTech]

Did you expand your subnet from before or just chop up an existing range?

[themightymrp]

Chopped up an existing range. We went from a 172.27.192.0/19 to a number of smaller ones ranging from /23 to /29

[IrritableTech]

Thinking out loud here... is there any port protection set up on your switches?

 

Or perhaps it's time to look at your switch config and document your dhcp scope options. Nothing should be sensitive, but feel free to PM if you prefer.

[themightymrp]

I will cobble that together and post it shortly. Just had some mild success with the following 3 steps on 2 machines. At least, it was a much quicker fix:

 

1) ipconfig /release

2) ipconfig /flushdns

3) ipconfig /renew

 

I need to try the same procedure on any more that pop up today. Does that point to anything? The DNS servers all passed the dcdiag tests. No port protections setup. The annoyance is that this only started after the VLAN implementation so there has to be something screwed somewhere... I just can't spot where.

 

I'll gather some configs and post them

[ADMaster]

Given you chopped up an existing range, I'd make sure all the new ranges have the correct subnet, and make sure you don't have anything using the old full range subnet mask.

 

This would include

DHcP scopes

vlan configs on the switch.

servers / printers with static addresses.

[themightymrp]

All of those checked and double checked. Can I ask anyone who has a Smoothwall how you have your DNS set up?

 

I have my 3 DNS servers set up with a forwarder address of the Smoothwall IP. Is that correct? It was how it was set pre-VLAN but with the new settings I wanted to be sure. The Smoothwall internal IP changed you see.

[themightymrp]

OK so attached are 2 files. One being the core switch config (nothing in there giving anything away). To explain a few parts:

 

1) The second ip-helper address listed is my WDS/MDT server. Needed that in there for BIOS/UEFI support

2) There are 2 static routes mentioned further down. One is the default 0.0.0.0 route which points at the Smoothwall. The other is a route for all internal IP addresses to go through the core switch interface 172.27.198.254 - I found I had to add a route like this to each switch other I couldn't ping them if I wasn't on the same VLAN as the switches. Could this be a sign of the problem???

3) The switch has the IP of our 3 internal DNS servers so that I can run ping commands from the switch by using names.

 

The second attachment is the route information contained on the Smoothwall. Getting this info added was thanks to Schools ICT as I couldn't get the internet to work.

 

All clients get DNS server settings of our internal servers. These servers have a forwarder set to be the IP of the Smoothwall. The Smoothwall has DNS forwarders of Googles DNS addresses.

 

Happy to answer any queries if you can help me!

Core switch config.txt

[IrritableTech]

Cheers - yeah the static route to the vlan 1 interface is a bit weird. It feels like it could be causing a bit of a loop and I can't see why it should be needed.

 

When you updated all the configs on your other switches, what did you set their default gateway to? They should ping just like every other device as long as they have a correct gateway to send the return packet to.

 

What is the switch utilization like?

 

I can't see anything else in your config which is jumping out at me.

[themightymrp]

I was finding I couldn't ping anything without it in, so I do believe that you are right in that there is a routing issue somewhere.

 

In order to be able to access each switch from any network PC I had to add a route on each switch of 172.27.192.0/19 --> 172.27.198.254 which is the IP address of the core switches VLAN1 interface. Once that was in I could connect to them all fine.

 

I was confused as to why the switch wasn't just doing the routing for me. But it is a WEIRD switch to configure so I wouldn't be surprised if I had something set wrong.

 

Attached is a cut down version of my network layout, quickly drawn. I don't like the extra static routes myself but without them it didn't like it. Combine the diagram with the static routes added to the Smoothwall (shown a couple of posts back) and that's the setup.

 

For the most part doing ipconfig /release, flushdns, renew in that order seems to work. But not 100% of the time

 

- - - Updated - - -

 

The DHCP server plugs directly into the core switch too. Its IP is 172.27.192.10

[IrritableTech]

For me the more straight forward way would be the following...

 

Core switch needs a default route pointing to 172.27.205.54 - the smoothwall

 

Each edge switch on VLAN 1 need a default route pointing to 172.27.198.254 - the core switch

 

Each vlan (with dhcp) needs an ip-helper to 172.27.192.10 - the dhcp server

And a default gateway setup to the appropriate IP on the core switch (much of this you already have).

 

And then your clients will get their default gateway from DHCP. They might attempt to ping a switch - but it's in another network so they send it to their Default gateway, the core switch routes the traffic to the correct edge switch. The edge switch send a reply to it's default gateway and the core routes the traffic and sends it back onto the client.

 

Having the ip route-static 172.27.192.0 19 172.27.198.254 on 172.27.198.254 suggests to me that there could be a loop - and if there is that could cause either a slow down, packets hitting their ttl or some port or loop protection kicking in and dropping the packets.

 

Our core has a ip route 0.0.0.0 0.0.0.0 10.10.1.254 - our internet suppliers router.

Our edge switches have a ip route 0.0.0.0 0.0.0.0 10.10.1.1 - our core switches IP within the 'switches' vlan.

All other settings are set via DHCP.

[themightymrp]

Hi Adam

 

Well I decided - what the hell - and deleted that second static route on the core switch just now. I did NOT lose any connectivity! That's a good thing. I will monitor for the next hour and see if I get any more errors crop up. If it turns out to be just that I'll be very happy.

 

The CPU and RAM utilization on the core switch I haven't seen go over 25% yet - ever!

 

The only thing I haven't done is change the static route on the outlying switches to be 0.0.0.0 0.0.0.0 to 172.27.198.254. It is still currently 172.27.19.0 255.255.224.0 to 172.27.198.254. But that might not make any difference.

 

Thanks for the help so far

[themightymrp]

Touch wood - so far so good. No new cases reported in last 45 minutes.....

[themightymrp]

OK not cured All looked good for a while there but it has started to occur again. Some of the clients are being really difficult to get connected again too.

 

Any more thoughts??

[themightymrp]

I'm going through all of the access switches and changing the static route on them to be a 0.0.0.0 0.0.0.0 to 172.27.198.254

 

Just to rule it out :-/

 

 

Update ---

 

Have now done the above. Will have to see if it makes any difference tomorrow.

 

Quick one - going by the Smoothwall route screenshot earlier in this thread, is it possible to replace all the separate routes with a single 172.27.192.0/19 --> 172.27.205.53 ?? Just to trim down possible errors and have one entry?

Edited February 1, 2018 by themightymrp

[IrritableTech]

I'm going through all of the access switches and changing the static route on them to be a 0.0.0.0 0.0.0.0 to 172.27.198.254

 

Just to rule it out :-/

 

 

Update ---

 

Have now done the above. Will have to see if it makes any difference tomorrow.

 

Quick one - going by the Smoothwall route screenshot earlier in this thread, is it possible to replace all the separate routes with a single 172.27.192.0/19 --> 172.27.205.53 ?? Just to trim down possible errors and have one entry?

 

Too early to ask how things are?

 

Theoretically you just need one route to your whole /19 network from your smoothwall. ‘Any of these addresses go via here’.

[themightymrp]

I might get round to trying that after school today.

 

No joys with the rest yet Not many failures so far this morning but still a few. I'm not sure what else to eliminate or check? It's so strange because they all pick up, without fail, a correct set of DHCP settings straight away. But then they can't connect to anything??! In a few cases I have been able to ping the default gateway, and indeed, every default gateway address stored on the core switch. But nothing within other VLANs or even within their own VLAN??

[themightymrp]

OK, interesting development.

 

We have a small number of computers in our PE block which are having the issue. 8 PC's in total, only numbers 4 and 5 working. I went onto the DHCP server and deleted the leases for numbers 1,2,3,6,7 and 8. Then went onto the DNS servers and deleted their DNS records for both forward and reverse lookup zones. Checked all 3 servers to make sure they all looked the same.

 

I've done a DHCP renew on the non-working machines and they have appeared on the DHCP leases on the server. However they haven't created DNS records? Only machines 4 and 5 have got their DNS records in place. Could there be an issue here do you think? How would I test it?

[themightymrp]

Attached is a picture of the DNS update settings tab on the DHCP server. Are these the default settings? I think they are. Is it worth switching the option to always dynamically update DNS records?

[IrritableTech]

Attached is a picture of the DNS update settings tab on the DHCP server. Are these the default settings? I think they are. Is it worth switching the option to always dynamically update DNS records?

 

Ours here is set like yours - client requests the update.

Your DHCP server is authorized isn't it? Any network access protections set? Is name protection switched on?

 

Any IPv6 setup on your network?

[themightymrp]

Right, watch this space. I may have something. Under the advanced tab of the DHCP server settings, there was a user account entered here for doing DHCP updates... an account that hasn't been used for years and I've no idea why it was there?! But it was configured and various other settings were not.

 

I've used the best practice guide found here :

 

https://social.technet.microsoft.com/Forums/windowsserver/en-US/9e765f5a-1656-4cde-809b-3c99c0fa78d2/enabling-user-account-on-dhcp-server-to-update-dns-records?forum=winserverNIS

 

And set things up slightly differently. DNS records are now being created and removed instantly which seems to be having the desired effect. I need to wait now for more issues - in fact I'll head to music as they were having problems with the Apple Macs!

 

Oh, name protection is not switched on. No IPv6 on the network however it is in the default enabled state on all machines.