NTP Server Issues

[CHiLL]

Over the past couple of days, our clients have been running 5 minutes fast. In my efforts to resolve this, I cannot figure out why the time is incorrect. Here is our NTP setup:

 

VMware Host 1 & 2: Each server is configured to use 0.uk.pool.ntp.org, 1.uk.pool.ntp.org, 2.uk.pool.ntp.org, 3.uk.pool.ntp.org

PDC (a VM): NTP not configured and uses the Local CMOS clock (which is the vmware host)

Non-PDC DC: NTP not configured and states source as PDC

Clients: NTP not configured and states source as PDC.

 

Despite that, the PDC is 5 minutes ahead. vSphere reports that both hosts have the correct time in the web interface. Despite pulling it's information from the host, the PDC is incorrect.

PS C:\WINDOWS\system32> w32tm /stripchart /computer:time.windows.com /samples:3 /dataonly
Tracking time.windows.com [51.145.123.29:123].
Collecting 3 samples.
The current time is 13/09/2022 10:01:06.
10:01:06, -183.7802435s
10:01:07, -183.5787555s
10:01:09, -183.3756424s

 

PS C:\WINDOWS\system32> w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 1 (primary reference - syncd by radio clock)
Precision: -23 (119.209ns per tick)
Root Delay: 0.0000000s
Root Dispersion: 10.0000000s
ReferenceId: 0x4C4F434C (source name:  "LOCL")
Last Successful Sync Time: 13/09/2022 09:35:42
Source: Local CMOS Clock
Poll Interval: 6 (64s)

 

Despite that, the host is currently reporting: 09/13/2022, 9:57:43 AM

 

It seems to be causing havok with other systems and I want to get this resolved as soon as possible!

[Davit2005]

I know in the past I have set VMWare hosts to look at external NTP and the PDC emulator that was VM on host to external ntp sources. Cannot remember exact cli command on Windows I used at the time was 6 years ago.

[tosh74]

w32tm /query /configuration

 

check that VMICtimeprovider is not listed as a time source. If so remove it

[CHiLL]

w32tm /query /configuration

 

check that VMICtimeprovider is not listed as a time source. If so remove it

It's set to (Local):

PS C:\WINDOWS\system32> w32tm /query /configuration
[Configuration]

EventLogFlags: 2 (Local)
AnnounceFlags: 10 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 6 (Local)
MaxPollInterval: 10 (Local)
MaxNegPhaseCorrection: 172800 (Local)
MaxPosPhaseCorrection: 172800 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 7 (Local)
UpdateInterval: 100 (Local)


[TimeProviders]

NtpClient (Local)
DllName: C:\WINDOWS\SYSTEM32\w32time.DLL (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
CrossSiteSyncFlags: 2 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Local)
ResolvePeerBackoffMaxTimes: 7 (Local)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 1 (Local)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 1024 (Local)
Type: NT5DS (Local)

NtpServer (Local)
DllName: C:\WINDOWS\SYSTEM32\w32time.DLL (Local)
Enabled: 1 (Local)
InputProvider: 0 (Local)
AllowNonstandardModeCombinations: 1 (Local)

VMICTimeProvider (Local)
DllName: C:\WINDOWS\System32\vmictimeprovider.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)

 

Now when I try the following command: w32tm /resync /force, I get the following error:

The computer did not resync because no time data was available.

Edited September 13, 2022 by CHiLL

[Koldov]

PDC (a VM): NTP not configured and uses the Local CMOS clock (which is the vmware host)

 

Could be wrong but I thought best practise was that as the PDC is the only one that is an authoritative time server for the domain, it should therefore be the only one pulling from an NTP source and obviously everything else gets the time from it.

 

Definitely do not rely on HOST/CMOS and in fact make sure it is disabled between the HOST > VM settings (we use Hyper-V).

[CHiLL]

Could be wrong but I thought best practise was that as the PDC is the only one that is an authoritative time server for the domain, it should therefore be the only one pulling from an NTP source and obviously everything else gets the time from it.

 

Definitely do not rely on HOST/CMOS and in fact make sure it is disabled between the HOST > VM settings (we use Hyper-V).

I've also tried specifying the UK NTP pools listed earlier in the GPO for the PDC, but I'm still having the same issue.

Edited September 13, 2022 by CHiLL

[Koldov]

VMware Host 1 & 2: Each server is configured to use 0.uk.pool.ntp.org, 1.uk.pool.ntp.org, 2.uk.pool.ntp.org, 3.uk.pool.ntp.org

PDC (a VM): NTP not configured and uses the Local CMOS clock (which is the vmware host)

 

What does the Host CMOS time say (is that only visible on a reboot - I'm guessing so)...? Could the VM Host and its CMOS clock be different...?

 

I can't understand why you are asking the Host OS to get the time from an NTP source (which sets the time for the Host OS) and then the VM to get the time from the Host's CMOS and it's blowing my mind a little...

 

Anyway, our PDC (VM) has the following lines in w32tm /query /configuration:

 

Type: NTP (Local)

NtpServer: uk.pool.ntp.org,0x8 (Local)

 

Also all domain clients show:

 

Type: NT5DS (Local)

 

Are your Hosts on the Domain? If not I understand why they are getting an external NTP source for their time (ours are and so still get the time from the PDC VM)?

 

I had a bit of trouble a while ago (changed PDC and so time server) and had to manually set on a couple of clients:

 

w32tm /config /syncfromflags:domhier /update

 

net stop w32time

 

net start w32time

 

But the PDC always gets its time from an NTP source:

 

w32tm /config /manualpeerlist:"uk.pool.ntp.org,0x8" /syncfromflags:manual /update

 

I don't use a GPO as there is only one PDC I just did it manually.

[CHiLL]

Thanks. I'm not sure what the best practice is. I had to rebuild one of the hosts last week and I don't know if that's had any knock on effect.

 

Each VM has this option set in it's configuration, under VMware Tools: Synchronize guest time with host. That's been enabled and the default since we had them installed back in 2015. Are you saying that should be disabled/unchecked on my VMs?

 

I have just ran the command you listed (w32tm /config /manualpeerlist:"uk.pool.ntp.org,0x8" /syncfromflags:manual /update) and the PDC is now showing the correct time. Syncronize guest time with host is currently disabled on the PDC VM, as I hadn't turned it back on before running that command.

 

Edit: Despite that, w32tm /query /status is still showing the local CMOS as the source:

PS C:\WINDOWS\system32> w32tm /query /status
Leap Indicator: 0(no warning)
Stratum: 1 (primary reference - syncd by radio clock)
Precision: -23 (119.209ns per tick)
Root Delay: 0.0000000s
Root Dispersion: 10.0000000s
ReferenceId: 0x4C4F434C (source name:  "LOCL")
Last Successful Sync Time: 13/09/2022 11:17:36
Source: Local CMOS Clock
Poll Interval: 6 (64s)

Edited September 13, 2022 by CHiLL

[Koldov]

Try this:

 

w32tm /config /manualpeerlist:uk.pool.ntp.org,0x8 /syncfromflags:manual /reliable:yes /update

 

You might also have to stop and restart the time service.

 

net stop w32time

net start w32time

 

Then:

 

w32tm.exe /resync /rediscover /nowait

 

 

 

I think it's going to be a little more complicated due to having a GPO undoing any manual set-up.

 

I've never actually looked up 'best practise' but have certainly read many times not to sync time with the host, as far as I'm aware the PDC gets its time from NTP and all other 'on-domain' computers (whether they are VM or not) get their time from the PDC and nowhere else, as if they were getting their time from CMOS or host they could be different.

 

There are a couple of posts on here about it:

 

http://www.edugeek.net/forums/windows-server-2019/228381-ntp-gpo-workstations-not-listening.html

 

http://www.edugeek.net/forums/windows-server-2012/223638-time-out-dc.html

 

http://www.edugeek.net/forums/windows-server-2019/217637-windows-time-out-hour.html

 

~A bit more complicated but might turn out to be time related...

 

http://www.edugeek.net/forums/windows-server-2016/220462-dc-issues.html

 

And more here:

 

https://community.spiceworks.com/topic/1682342-time-sync-best-practices-hyper-v

 

EDIT: In w32tm /query /configuration the PDC should show 'Type: NTP (Local)' as above and every other 'on-domain' computer should show Type: 'NT5DS (Local)'.

 

EDIT2: Then in w32tm /query /peers the PDC should show the NTP server and all other 'on-domain' computers should show the PDC.

Edited September 13, 2022 by Koldov

[CHiLL]

So this has cropped it's ugly head again and I can't figure out why.

 

DC2 is our on-site NTP server, which syncs from uk.pool.ntp.org

w32tm /query /peers on DC1 points to DC2

w32tm /query /peers on other servers/clients reports either DC2 or DC1 as their source, using NT5DS.

w32tm /query /configuration on DC2 reports Type: NTP (Local) and The time service has started advertising as a time source.

w32tm /query /configuration on DC1 and other servers/clients reports Type: NT5DS (Local)

 

This morning, DC2 was correct but DC1 had gone forward 20 minutes and I have absolutely no idea why, because it's reporting that it's getting it's time from DC2.

[psydii]

Has DC01 rebooted? Is it on a different host to DC02? You may have bad bios clock on one of your hosts.

 

Upon reboot, a VM's clock is set to that of the host bios. If this has drifted far into the future, this might cause a problem for a DC because rolling back its clock can cause serious synchronisation problems in AD DS.

 

https://blogs.vmware.com/apps/2020/09/ensuring-accurate-time-keeping-in-virtualized-active-directory-infrastructure.html

 

https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/how-the-windows-time-service-works

 

https://dirteam.com/sander/2021/01/25/howto-configure-accurate-time-in-active-directory/

 

And this one might brute-force a fix for you:

https://dirteam.com/sander/2021/01/25/howto-configure-accurate-time-in-active-directory/

[CHiLL]

Has DC01 rebooted? Is it on a different host to DC02? You may have bad bios clock on one of your hosts.

 

Upon reboot, a VM's clock is set to that of the host bios. If this has drifted far into the future, this might cause a problem for a DC because rolling back its clock can cause serious synchronisation problems in AD DS.

 

https://blogs.vmware.com/apps/2020/09/ensuring-accurate-time-keeping-in-virtualized-active-directory-infrastructure.html

 

https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/how-the-windows-time-service-works

 

https://dirteam.com/sander/2021/01/25/howto-configure-accurate-time-in-active-directory/

 

And this one might brute-force a fix for you:

https://dirteam.com/sander/2021/01/25/howto-configure-accurate-time-in-active-directory/

Thanks, though DC1 is a physical box, not virtualised. It's running on HP's ILO4 and I can see that the time is correct in the ILO interface. I'll check out those links.

[CHiLL]

So looking through DC2's logs, I can see that VMware Tools was actually changing the time, which matches the same time difference we were seeing:

The system time has changed to ‎2022‎-‎09‎-‎21T11:54:34.475000000Z from ‎2022‎-‎09‎-‎21T11:40:45.096028100Z.

Change Reason: An application or system component changed the time.
Process: '\Device\HarddiskVolume2\Program Files\VMware\VMware Tools\vmtoolsd.exe' (PID 3608).

 

This is despite the setting "Synchronize guest time with host" most definately unchecked in the VM's "VM Options" settings tab. I don't know how many times I've checked myself and my colleague has also confirmed it numerous times.

 

DC2 is currently running from the host I rebuilt yesterday, so I clearly forgot to check the time settings on the host. But it doesn't explain why it was changing the time on the VM when the option was turned off.

 

I've now set the host to use DC2 and uk.pool.ntp.org as it's time servers and I hope that will clear up the issue.