How would YOU handle this situation? (DC Replication Problem)

[Ambient]

Here's the situation.

 

You have 3 domain controllers

 

Server1 (PDC, all FSMO Roles)

Server2 (Just an AD Replication Server)

Server3 (Exchange 2003)

 

It appears that Server 1 and 3 are happily replicating to one another, however, Server2 doesn’t want to know. Policy changes made on either Server 1 or 3 are replicated to one another, but Server2 isn’t replicating with them...thus creating policy discrepancies.

 

Users who are logged on to the domain by Server2 receive out of date policies, and computers that receive their computer policy from Server2 also receive out of date policies.

 

Server2 by the way houses the staff home folders and roaming profiles.

 

This problem seems to have started this weekend, perhaps due to the fact that I removed some old and un-used policies via GPMC. A re-boot of all the servers has been done, but it didn’t solve anything.

 

I think that pretty much covers the situation....how would you tackle it?

 

In my head, i pretty much know what the next step i want to takw is...but i want to know what you guys think first. Many thanks.

 

DCDIAG and FRSDIAG has been ran on the PDC, and the results can be seen in this thread - http://www.edugeek.net/forums/windows-server-2000-2003/57677-some-computer-not-updating-their-computer-policy.html

[irsprint84]

Gone into active directory sites and services?

 

on server2 try cmd -> NET TIME /DOMAIN:domain.local /SET

[PiqueABoo]

So sysvol isn't replicating to server two.

First place to look is in the FRS event logs (on all of the servers) and see what they're complaining about.

[Ambient]

Interistingly enough, when i do a manual replication in sites and services, every server replicates with the other two....with no errors. However, i've literally just checked the size of the "policies" folder of each DC. Server 1 and 3 have the exact same amount of files and folders in the policies folder...however, server2 has over 100MB more information in it (which is odd)

[Ambient]

Is that the same doing the FRSDiag?

[srochford]

Is any replication happening (eg if you change a user's details does that replicate) - you can check by targeting AD Users and Computers on server 1, making a change and then targeting at server 2 and seeing if the change has gone through.

 

Run dcdiag to see what that says.

 

If all else fails it's quick and easy to just dcpromo server 2 back to a standalone server, wait about an hour for things to settle (probably not necessary with only 1 other DC but if you have a multi-site setup you must wait long enough for all sites to synch) and then dcpromo back.

 

The only time that this is a bad idea is when you have Exchange on a DC - it will break (and I think SQL server also breaks if it's on a DC which you dcpromo down/up) but from what you say there should be no problem.

[Ambient]

dcpromo was what i was thinking...but i didnt want to mention it until someone else did. My plan is to do what i can to try and fix it during work tomorrow, and if i get no where...i'll demote and then promote.

 

Edit - but i wanted to see what people here thought was the best step. I'll try anything before i do the dcpromo, so please keep the ideas coming

[Ambient]

Gone into active directory sites and services?

 

on server2 try cmd -> NET TIME /DOMAIN:domain.local /SET

 

CMD isnt a strong point of mine, and i arent sure of the code i need to use to do this.

 

The FQN for our domain is cooper.intranet, and Server2 is actually called Cooperdata. Could you help me out with the exact code and switches?

 

Many many thanks

[PiqueABoo]

No relevant errors anywhere..you sure?

Next thing I might do if that really is the case is non-authoritative NTFRS recovery on server 2, backing up sysvol and deleting it's contents (*not* the folders) in the middle.

Google for NTFRS and JOURNAL WRAP for info on this - the trick is to arrange it so that server 2 will get fresh sysvol contents from one of the two "good" DCs, as opposed to the other way around.

[Ambient]

This error exists on Server2.....

 

The File Replication Service has detected that the replica set "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)" is in JRNL_WRAP_ERROR.

 

Replica set name is : "DOMAIN SYSTEM VOLUME (SYSVOL SHARE)"

Replica root path is : "c:\windows\sysvol\domain"

Replica root volume is : "\\.\C:"

A Replica set hits JRNL_WRAP_ERROR when the record that it is trying to read from the NTFS USN journal is not found. This can occur because of one of the following reasons.

 

[1] Volume "\\.\C:" has been formatted.

[2] The NTFS USN journal on volume "\\.\C:" has been deleted.

[3] The NTFS USN journal on volume "\\.\C:" has been truncated. Chkdsk can truncate the journal if it finds corrupt entries at the end of the journal.

[4] File Replication Service was not running on this computer for a long time.

[5] File Replication Service could not keep up with the rate of Disk IO activity on "\\.\C:".

Setting the "Enable Journal Wrap Automatic Restore" registry parameter to 1 will cause the following recovery steps to be taken to automatically recover from this error state.

[1] At the first poll, which will occur in 5 minutes, this computer will be deleted from the replica set. If you do not want to wait 5 minutes, then run "net stop ntfrs" followed by "net start ntfrs" to restart the File Replication Service.

[2] At the poll following the deletion this computer will be re-added to the replica set. The re-addition will trigger a full tree sync for the replica set.

 

WARNING: During the recovery process data in the replica tree may be unavailable. You should reset the registry parameter described above to 0 to prevent automatic recovery from making the data unexpectedly unavailable if this error condition occurs again.

 

To change this registry parameter, run regedit.

 

Click on Start, Run and type regedit.

 

Expand HKEY_LOCAL_MACHINE.

Click down the key path:

"System\CurrentControlSet\Services\NtFrs\Parameters"

Double click on the value name

"Enable Journal Wrap Automatic Restore"

and update the value.

 

If the value name is not present you may add it with the New->DWORD Value function under the Edit Menu item. Type the value name exactly as shown above.

 

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

[PiqueABoo]

That explains everything - thought it might be that - and it is not too hard to fix.

 

If you're paranoid then go to Server1 *or* Server3 and make a copy of the good sysvol there (with Explorer to some nearby temp folder).

 

Then do what the event says with that reg value on Server2 - it performs a non-authoritative restore i.e. Server2 will get the latest, greatest sysvol contents from one of the other servers.

 

Do set the reg value to 0 afterwards as instructed.

[Ambient]

Ace, i'll give that a go.

 

I soooooo want to crack on with this job now from home, however, if something goes wrong, i wont be able to sleep tonight without getting in my car and going to work at this hour lol. So, i'm going to do this first thing in morning.

 

Is this something you've personally come across before by any chance?

[Ambient]

One more thing, and i'm going to sound like a TOTAL n00b when i say this. When i read that guide in order to fix this, there are a number of words that i quite simply dont understand in their subjective context. Typical examples of this are "poll" and "replica set".

 

I dont suppose anyone would be so kind so simplify the steps in that "guide" for me?

[Ambient]

I couldnt help myself....so i've started the repair. It seems to be going through it alright. Is it a long job does anyone know?

 

Also, since the job started, the Sysvol folder on the server has gone (not literally...but it isnt showing up as a share anymore), will this come back automatically when the process finishes?

[PiqueABoo]

The share does go away so it can't get used whilst it's being recreated, but should come back automatically and IME it should have all finished now.

[Ambient]

Aye, all finished, and all working again now. Words can't describe how much of a weight off my shoulders this is. Many thanks for the help

 

In the "new" sysvol folder, there is a folder called "Ntfrs Pre-Existing......see event log"....do i need this...or should i delete it?

[srochford]

You almost certainly don't need it.

 

It's a copy of what was previously in sysvol on that server. That ought to be exactly the same as on the other server but replication wasn't working properly so it's not the same - the folder is kept in case there's anything on server2 that's not on server1. If you're pretty sure that replication was OK when you last made policy changes then just delete it (and even if you're not sure you can delete it; there's no easy way to get back the stuff in a usable form!)

 

The time command was shown wrongly in the example above - it should just be:

net time /domain /set

- you don't specify the name of the domain at all (it doesn't make sense to give a domain name; what you're saying is "find the time source for my domain and set this machine to match"). You only need to do it if there's a time error (w32tm) in the event log. (Incidentally, Server2003 and later support the w32tm command - you just type:

w32tm /resync

 

Finally, don't be afraid of dcpromo - provided that you've got at least one good working server as well as the one you're demoting there should be no problem. If you are worried, bring up another server (it can be virtual), dcpromo it so it is a DC and let it replicate. Once that's working OK you know you've got 2 working DCs and you can safely demote the other one.

 

Good to hear it's all OK now :-)

[PiqueABoo]

I also think it's a copy of the files that were in sysvol on Server2, essentially a backup made before they were replaced by a copy from one of the other DCs.

 

Unless it is really big (why though?) and you don't have disk space, then being paranoid I would leave it alone until you're sure everything is fine and have done a server backup or two.

 

there's no easy way to get back the stuff in a usable form!

 

I suppose you have to know what you're doing, but it's not that hard. Think this is more a case of MS wanting to minimise the messes careless people could make when fiddling, so they've left it all sounding a bit arcane and scary.

[Ambient]

Fair point, hard drive space is far from an issue on this server, so i think i'll just leave it well alone.

 

Thanks again for everyones help with this issue, you're all great.

[bio]

Good to see its fixed... a good tool to play with and troubleshoot replication issues is m$ replmon : Replmon Overview: Active Directory

 

regards

bio..