AD time is 1 hour behind this morning - what will break when I fix it?

[AngryTechnician]

Noticed this morning that the time across my entire network is 1 hour behind. The clock on the FSMO DC has gotten behind somehow, and of course everything syncs with that by design.

 

Everything is working normally other than that, and I'm looking into why it has happened now, but my main concern is what could break if I resync the DCs now. I know AD replication relies on the time delta being less than 15 minutes, so I can sync those manually, but will the workstations carry on working while they catch up?

 

We have no 802.1x in operation so no worries there.

 

Situation so far:

• The FSMO DC syncs with time.windows.com and is set as the reliable time source for the AD. - CHECKED OK
   
  
• All other servers and workstations sync with the FSMO. - CHECKED OK
   
  
• Time sync between host and guests in Hyper-V is disabled for all DCs as per best practice - CHECKED OK
   
  
• Time zone is set correctly (to "(UTC) Dublin, Edinburgh, Lisbon, London") and daylight savings adjustments checked on the DCs and workstations.
   
  
• I have a couple of errors on the FSMO for Time-Service from yesterday morning at about 3am saying that time.windows.com was unreachable, but no other fault since.
  

  Time Provider NtpClient: No valid response has been received from manually configured peer time.windows.com after 8 attempts to contact it. This peer will be discarded as a time source and NtpClient will attempt to discover a new peer with this DNS name. The error was: The peer is unreachable.

• All DCs and Hyper-V hosts were rebooted to install updates yesterday morning (Monday) at about 3am, as per their scheduled update settings. I have a sneaking suspicion that whatever happened, happened then, but I'm not sure exactly what.
  

Edited May 3, 2011 by AngryTechnician

[teejay]

Anything using Kerberos will break if it's more than 5 minutes out, so all your clients will fail authentication until they sync time properly.

[AngryTechnician]

Anything using Kerberos will break if it's more than 5 minutes out, so all your clients will fail authentication until they sync time properly.

Meaning no new logons, correct?

[teejay]

Meaning anyone logged in trying to access a share or anyything will fail as well until the time syncs properly.

[glennda]

Yeah i would wait until tonight remote desktop to all your severs then set it on the PDC and run net time /set on the others and the clients should update when booted (in theory) there was only 2 clients which didn't here out of 800 when i corrected although mine was only 4 mins out.

 

Toby

[AngryTechnician]

OK, so as per my original gut response, I won't be fixing this until this evening. Still can't find any smoking gun to indicate what's caused it though.

[teejay]

Yeah i would wait until tonight remote desktop to all your severs then set it on the PDC and run net time /set on the others and the clients should update when booted (in theory) there was only 2 clients which didn't here out of 800 when i corrected although mine was only 4 mins out.

 

Toby

 

Apart from you may lose your vpn connection or remote desktop connections when you change time on the server. Best done on site rather than remotely ;-)

[DrCheese]

Sure it's not just the timezone set incorrectly? I noticed this a few weeks ago when some of our Windows 7 clients were set to "Greenwich Standard Time(Iceland)" instead of "GMT(London)"

You can change the timezone without Windows getting in a fizz, but if you change it by hand you'll end up with the 5 minute issue

[AngryTechnician]

Checked the timezone on workstations and the FSMO, both set correctly.

 

I will be onsite when I fix this, not taking any chances there.

[psydii]

Kerberos will break until all the clocks are back in sync. So logons will fail. Other than that depending on you're set up the users *might* not see anything else as Windows will fall back to NTLMv2.

 

What are your DC's syncing to? Are they in a VM by anychance?

 

W2K used to have setting to controll how far a clock could jump at anyone time if it was out of sync with its time server. Technet implies this no longer works, but I could have read that wrong. To be safe I'd perhaps set a script to knock the time forward by 4m30s (my recollection is that kerberos tickets default MaxClockSkew is 5 minutes) every hour for 13hrs and then ensure the DCs are syncing to a reasonably reliable external source (e.g. your ISP time server or uk.pool.ntp.org)

 

An hour isn't too bad, but DC's that get way out of time are very very bad news. Here's a horror story: Leonid's notes: Wrong Time on a Domain Controller

 

As general background reading, this may be useful:How to configure the Windows Time service against a large time offset

Edited May 3, 2011 by psydii

[chazzy2501]

I noticed that the time on my Joggler (WifI Picture Frame, that uses npt micrsoft) was an hour behind this weekend (Also had the wrong month for a couple of hours?!)

[psydii]

Just spotted your update up top. If they've stopped syncing to the external time source for some reason and exist in a VM you're lucky they are only an hour out.

 

I'd warn your users not to trust the time, and then gradually bring them back into sync. While you're doing that investigate what might have caused the sync to fail. I believe that reconfiguring via w32tm and a restart of the time service should bring external synching back to life for you, but I wouldn't do that until you are sure that your domain and users are within 5 minutes of the correct time, or you have confirmed that the DC will automatically limit themselves to carefully paced sub five minute corrections, so as not to loose or cause errors in kerberos authentication.

 

Or you could wait for the evening, correct the time and reboot everything that's not the authoritative time source for the domain? The flaw in my original suggestion is that being in VMs any correction may have drifted by the time the script next runs.

 

As for root cause: your servers all restarted around 3am.. is it possible the the DC that was trying to contact its external time source couldn't get past the firewall?

 

Further, thought on root cause: The VMs BIOS initially gets it time from the Host, so if NTP has failed, your DC has nowhere to get its time but from the vm bios. What are the timezone settings on the VM hosts?

Edited May 3, 2011 by psydii

[AngryTechnician]

Timezone settings on the VM hosts are correct, and as stated above, the time sync integration service is disabled in Hyper-V for all DCs anyway specifically to stop time creep.

[irsprint84]

Some commands that helped me with problems:

 

w32tm /query /status - do on client ot see where its getting time from

w32tm /resync /rediscover - this should point clients to dcs

 

Also use /status to see where dcs are getting time from if CMOS I advise resetting to a internet time

[AngryTechnician]

OK, the servers and workstations I've checked are getting their time from DCs as they should be. However, the FSMO is returning the following:

 

Source: Free-running System Clock

 

which is clearly Not Right. So it looks like external time sync has failed on the FSMO. The question is why it then fell back to an incorrect time. This server is a VM, but without the time sync integration, where would the CMOS time come from?

 

Also, I will be giving up on time.windows.com from now on as I've found a slew of intermittent failures from there going back months. Planning to use europe.pool.ntp.org unless anyone has anything bad to say about it?

Edited May 3, 2011 by AngryTechnician

[glennda]

which is clearly Not Right. So it looks like external time sync has failed on the FSMO. The question is why it then fell back to an incorrect time. This server is a VM, but without the time sync integration, where would the CMOS time come from?

 

Will it not fall back to the time set in the bios? or just generally work its way out of sync slowly (vm's tend not to be great sometimes with the time

[AngryTechnician]

I think it must have been slowly working it's way out of sync since the internet time server stopped working. Over a long weekend while I wasn't monitoring it I could easily see it build up an hour. Will monitor the exact delta over the rest of the day and see if it creeps any more.

[irsprint84]

CMOS isnt reliable my servers were doing wierd things when it looked at CMOS so just point to uk.ntp servers then send resync command to clients

[psydii]

Sorry, I understood that Sync is disabled, but the Virtual BIOS has to get the initial time (during its 'power-on') from somewhere. In this case the initial seed from the host has the correct time settings, so either way can't be the root of the problem.

 

Just out of interest is it exactly or 'about' one hour out? <<-nevermind, answered already!

 

VMs cannot hold time. this is why they must sync with an external source, and why the Time Sync Service exisits, however as you have done, Best practice for domain environments is to disable that and use NTP instead. If connection to NTP is lost then the PDC emulator falls back to 'clock ticks' which are generally fine when running on bare metal, but utterly unreliable inside a Virtualised environment.

 

VMWare based but the prinicples hold: http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf

Edited May 3, 2011 by psydii
too slow, question answered already.

[AngryTechnician]

Right now it is behind by 52 minutes and 25 seconds, using BST: British Summer Time: UK Daylight Saving Time (GMT +1) as the current time source.

[glennda]

If you are part of an LEA you may find that some external times sources are blocked. I know that I have to use our LEA's timeserver as we cannot get out on port 123(i think its that)

[AngryTechnician]

We're independent, our Internet connection is a raw VDSL feed from Zen. Port 123 is not blocked on our router.

[chazzy2501]

you could buy one of these:) (Always wanted one)

 

Rackmount GPS NTP Server | NTS-4000-GPS-R | Galleon Systems Ltd.

[AngryTechnician]

Only $1500! BARGAIN!

 

Now I think about it, I wonder why all our phones that have GPS built in can't sync time from it? My Android can only sync from NTP or the mobile network, but the data is clearly available because I can see it in the GPS Essentials app.

[teejay]

Use uk.pool.ntp.org rather than europe.pool.ntp.org