HELP ! Replication issue

[fiza]

Came in this morning and lots of users unable to log in, some could but had no mapped drives and no printing.

After some investigation I thought I had an issue with one of the DCs so moved the FSMO roles it had to another DC and demoted it.

This meant I had 2 DCs left but now DC1 will not talk to DC2.

 

DCDiag on either server brings up "DC is not responding to directory service requests" for Default-First-Site-Name test.

All other tests pass.

 

On DC2 when I look at operations masters it shows current holder of RID,PDC and Infrastructure as "Error" so cannot see that DC1 has the roles.

On DC1 operations master shows correctly that it has the

 

When I go to "Active Directory Sites and Services on and try to right click to "Replicate now" I get;

 

The following error occurred during the attempt to contact the Domain Controller DC1: The RPC server is unavailable.

 

 

 

Any help would be appreciated!

[Oaktech]

Are your update and patches consistent across all DCs and member servers?

[fiza]

Are your update and patches consistent across all DCs and member servers?

We do update all DCs at the same time so I would hope so.

[HPlum78]

A few questions here:

 

How many DC's do you have?

What OS are they running?

How did you move the FSMO role and are you sure that it was successful? - can you run Get-ADForest | Format-Table SchemaMaster,DomainNamingMaster and then Get-ADDomain | format-table PDCEmulator,RIDMaster,InfrastructureMaster

(or netdom query fsmo) I dont care what command is used TBH.

Edited May 15, 2023 by HPlum78

[HPlum78]

oh and just out of interest is there any newish events in the directory service logs with the source of NTDS KCC? (clearly there will be some as you have demoted a DC but just wondering if there have been any others of note?)

[robintech]

Does all the DNS resolution work for themselves and each other , they're not going somewhere else for it ?

[djm968]

Look at the simple things first, did the demoted Dc or has the other DC run out of diskspace, sounds silly but I've missed this before in a blind panic to try to get things working again as quickly as possible and actually made the issue worse!

[fiza]

A few questions here:

 

How many DC's do you have?

What OS are they running?

How did you move the FSMO role and are you sure that it was successful? - can you run Get-ADForest | Format-Table SchemaMaster,DomainNamingMaster and then Get-ADDomain | format-table PDCEmulator,RIDMaster,InfrastructureMaster

(or netdom query fsmo) I dont care what command is used TBH.

 

DC's were 3 now 2

All running Server 2016

 

I demoted the server by going into Active Directory users and computers> Domain Controllers and right clicking on the server to be deleted and ran through the steps.

 

If I run netdom query fsmo on DC1 then it tells me correctly that Schema Master and Domain naming master are held by DC2 and DC1 holds PDC,RID and Infrastructure.

If I run netdom query fsmo on DC2 then I can see it holds Schema Master and Domain naming master. for PDC, RID and Infrastructure its shows "Warning: role owner is a deleted DC"

 

IS DC2 the issue? Did I make a mistake thinking it was the now deleted DC that was the issue?

 

- - - Updated - - -

 

Look at the simple things first, did the demoted Dc or has the other DC run out of diskspace, sounds silly but I've missed this before in a blind panic to try to get things working again as quickly as possible and actually made the issue worse!

 

Space on the servers does not seem to be the issue as they have plenty.

[fiza]

In the Event viewer for DC2 I am seeing lots of the same errors ;

"ServerAgent::getDomnsInfo(), ADsGetObject() (local machine) failed, hr = -2147016672 (0x80072020"

and

"ServerAgent::refreshMachineSidsAndDomains().failed to query active directory for domains info"

[HPlum78]

So did DC3 hold any FSMO roles prior to it being removed from AD?

[fiza]

Trying to load DNS on either server I get "The server DC* coould not be contacted. The error was: Access was denied. Would you like to add it anyway".

where * is the the DC I am trying from.

[fiza]

So did DC3 hold any FSMO roles prior to it being removed from AD?

 

Yes , PDC, RID and Infrastructure but during the demotion process which I carried out on DC1 the roles supposedly transfered successfully to DC1.

DC2 is still thinking that DC3 is the holder and is a deleted server.

[HPlum78]

https://www.microsoft.com/en-gb/download/details.aspx?id=30005 - is the GUI based repadmin tool essentually, if you can download it and run it.

[djm968]

You might need to manually force demote DC3 and seize the roles.

https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-manually-removing-a-domain-controller-server/ba-p/280564

https://learn.microsoft.com/en-us/troubleshoot/windows-server/identity/transfer-or-seize-operation-master-roles-in-ad-ds

https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/demoting-domain-controllers-and-domains--level-200-

Edited May 15, 2023 by djm968

[HPlum78]

So @djm968 is probably right but the issue here is that I dont think that replication is currently working so where you run these commands from will matter. If replication is an issue then just blindly removing meta data and grabbing roles will cause you more issues!

[HPlum78]

So what I am trying to say if anything, understand where you are right now before doing anything that is going to change the underlying AD environment especially if those changes are going to rely on the underlying replication topology to update the other DC.

[fiza]

https://www.microsoft.com/en-gb/download/details.aspx?id=30005 - is the GUI based repadmin tool essentually, if you can download it and run it.

Unfortunately when install and run the GUI based tool is tells me "This version of the tool has expired. Please download the latest version from https://aka/ms/adreplicationstatus"

Downloading and installing from there does nothing different and I get the same message.

 

- - - Updated - - -

 

Seems there is an issue with this tool and lots of people complaining they have the same problem!

[HPlum78]

oh the joy, So DNS lets try to get access to that,

 

if you open up the host file on one of the DC's (c:\windows\system32\drivers\etc\) - use notepad and will probably need to be an elevated an all. (take a copy of the original)

 

Add the following to it:

 

127.0.0.1 localhost

::1 localhost

 

then try yo open dnsmgmt.msc

 

See if that lets you get in, if not there are other things we can try.

[fiza]

I have access to DNS now but only one the Server I am on. If I try to add the other server I get

"A security package specific error occurred. Would you like to add it anyway?"

[fiza]

So from what i can decipher I have one DC that seems to be having issues and I may have demoted the wrong one thinking originally that the issue was there.

 

DC1 can run netdom query fsmo successfully and knows what roles it has and what roles the other DC has.

DC2 can run netdom query fsmo and shows itself as an operations master for Schema and Domain naming but shows the other roles as belonging to the demoted server.

 

On DC1 when I view operations masters in User and Computers it shows itself as the master.

 

When I view operations masters in Domains and Trusts it shows DC2 as the operations master but the 'change' option is greyed out and it shows "The current operations master is offline. The role cannot be transferred."

 

On DC1 when I view operations masters in Users and Computers it shows "ERROR" where the master should be.

 

On DC2 Domains and Trusts does not load at all showing "The configuration information describing this enterprise is not available. The target principle name is incorrect"

 

 

Is DC2 the culprit?

[HPlum78]

so before i will pin it on either DC, can you do the same on the other DC with regards to host files to get DNSmgmt working, can you check the settings on of DNS on each server and check that it is pointing to itself secondary (ip address) and then to the other DC's address (ip address) as the primary (in the network configuration). Hope that makes sense.... Edited May 15, 2023 by HPlum78

[fiza]

so before i will pin it on either DC, can you do the same on the other DC with regards to host files to get DNSmgmt working, can you check the settings on of DNS on each server and check that it is pointing to itself primary (ip address) and then to the other DC's address (ip address) as the secondary (in the network configuration). Hope that makes sense....

 

I have DNS management working on both servers now. I am not sure I understand how to check if the server is pointing to itself as primary?

[djm968]

Check the IP config on NICs of both servers.

 

Primary DNS of DC1 should be set to the IP of DC1. Secondary DNS should be IP of DC2

 

Primary DNS of DC2 should be set to the IP of DC2. Secondary DNS should be IP of DC1

Edited May 15, 2023 by djm968

[HPlum78]

Sorry i have just edited my initial post and changed the order, either way right now just want to make sure that the servers are not trying to point to a DC that is no longer available, so as @djm968 says just to try and get this back up and running.

 

We can deal with the what points to what and replication islands and all that later....

Edited May 15, 2023 by HPlum78

[fiza]

Check the IP config on NICs of both servers.

 

Primary DNS of DC1 should be set to the IP of DC1. Secondary DNS should be IP of DC2

 

Primary DNS of DC2 should be set to the IP of DC2. Secondary DNS should be IP of DC1

 

Both showing correctly.