DHCP Broken!

[Sheridan]

This last couple of days has been a nightmare.

 

A power cut knocked some switches off, and (possibly coincidentally) since then dhcp has been behaving very oddly. We run dhcp on a server 2012 box, and its working in an odd way now. Certain pcs just won't get an address, even though a pc in the port next to the 'failing' one works ok.

 

We have several subnets linked at a layer 3 switch, with the DHCP helper option configured for the server, this has been working fine for years and only now seems to be a problem. When I check the dhcp logs at the server I can see loads of NACK errors for the failing pc:

 

15,10/24/14,14:15:31,NACK,192.168.37.5,.domain.name,,,0,6,,,,,,,,

15,10/24/14,14:15:31,NACK,192.168.37.5,,,,0,6,,,,,,,,

 

I've even replaced the switches in the relevant area, but this seems to be random. Different PCs on different subnets are getting the same error without any logical kind of pattern. I've totally removed dhcp and rebuilt it from an earlier backup but the problems still persist. Its going to develop into a major headache after the holidays if I don't find out whats going wrong!

 

The fact that the server is showing loads of NACK messages in the logs means that the DHCP helper option seems to be working as the server is receiving the DHCP requests, but why does it continually reject the requests for certain pcs, and why all of a sudden?

Edited October 24, 2014 by Sheridan

[truebluesteve]

Are the problem computers getting any IP at all. Although it's seems unlikely given what you have said, I wonder if there is another devices acting as a DHCP server on your network?

[Joanne]

could it be that a switch config wasn't saved? Then because it has been switched off, all configuration on it is lost? Could even be a single port that was changed or something?

[win]

Can you reset the switch back to factory defaults?

[Sheridan]

I reset the relevant edge switche back to default, by doing a firmware re-flash. The thing is, these failing PCs seems to eventually get an ip address after maybe 10-15 minutes - during which time the server is showing a stream of NACK messages in the logs.

 

I did think of a rogue dhcp server - but the actually dhcp server is effectively showing the the requests are going to the right place, but just get denied for a long time.

[win]

I would try creating a new DCHP service on a different DC, then decommission the existing DHCP service.

 

if you have enough DCs then do a split scope which might help with troubleshooting.

[pantscat]

Have you definitely got enough addresses left in the scope? Or perhaps a dodgy Mac table on a switch?

[Dickens]

Start by narrowing down the problem, take away the stuff that is just noise and leave yourself with something easy to troubleshoot. Take two PCs one that gets an address and one that doesn't then swap their ports on the switch - does the problem follow the PC or stay with the port ? Do you run VLANs across the network - could the loss of power have reverted a port based VLAN to the default VLAN or vice versa. It does sound as though it could have been switch config in running memory not saved to NVRAM - been there, done that. IS this on one particular switch or multiple switches, are you able to directly connect a couple of PCs to the same switch as the server and if so, do they both get addresses ?

 

You could try installing WireShark on a PC, monitor the network traffic then switch off your DHCP server, see what traffic you still have - this should highlight any rogue DHCP servers that are causing trouble.

[Sheridan]

All the ranges were around 10-15% full, so loads of addresses.

 

It was part of a two server setup (failover) but I've removed it from both, and recreated it on one to be sure. I'm wondering if its just Server 2012 as my previous 2003 dhcp server ran for 10 years without a glitch on the same network.

[Sheridan]

I'm going to have to just ditch dhcp on server 2012 as it seems to be pretty buggy. Has anyone managed to move dhcp from 2012 back to 2008? Not the usual migration but I've wasted days on this and its going to be a major issue. The network topology/switching hasn't changed since dhcp was originally migrated from 2003 to 2012 so I'm starting to think the power cut timing was pure coincidence!

[ful56_uk]

how you get on with this issue, i found over the weekend one of my scopes has decided to stop giving out ips, i have 8 scopes and my hp core l3 switch is the ip helper hasn't changed either

i see a few back 0 6 in the log file for dhcp, its really strange

[Sheridan]

My problem seemed to be a really obscure one in the end. It came down to some of our wireless APs forming a mesh across two subnets - which meant that some devices were (I think!) simultaneously requesting an IP address from two ranges - the NACK entries in the log were effectively correct as the renew request for the IP address was coming from the the wrong subnet!

 

I went through every AP and made sure the meshing was off/manually configured and everything settled down to normal behaviour. Why our Ruckus system decided to do this is beyond me, but it took some finding!

[ful56_uk]

Shame we don't have ruckus. I'll keep plugging away

[Sheridan]

Shame we don't have ruckus. I'll keep plugging away

 

Might be worth trying the suggestions above - I did rebuild my DHCP server several times (after taking a backup!) and found that adding a failover server seemed to cause odd errors, so I went back to a simple single DHCP server on the original Server 2012 host.

 

I also made sure every switch still had STP enabled - 1 couple (out of 14 or so) needed it re-enabling after the power cut, I don't think it was relevant in my case but could be for you.