DPM 2012 - Replica Inconsistant

[Marshall_IT]

Hi,

 

I've been racking my brain and hitting Google hard the past week trying to fix issues with my DPM 2012 replicas.

 

I get 2 types of error occurring on a regular basis and on some of the items I cannot resolve this at the moment.

 

I get "Replica Inconsistent" and "Recovery Point Creation failed".

 

This is happening on SQL protection group and on a files share protection group.

 

However i have a mixture of OK and replica inconsistent on the same SQL instance.

 

Please help, pulling hair out

[adamf]

What is the actual error message?

 

When the job fails goto Monitoring, under Jobs, Failed Jobs for Yesterday and Today. Find the failed job and copy/paste the error.

 

Also, is it RTM, SP1, R2?

[Marshall_IT]

Synchronisation

Failed

 

The replica on volume... It's not consistent with the protection data source. Dpm error = 112 (Id 33-23)

 

Recovery point

Failed

 

Unexpected error occurred on dpm server machine during vss operation. I'd, 207

 

Looks like it might be to do with vss on the dpm server.

Edited February 11, 2014 by Marshall_IT

[adamf]

I had a similar issue with SQL databases where 5 out of 13 on the same instance would not create recovery points. Never really figured out what was causing it, but I ended up offsetting the sync time of the replicas and changing some of the TCP global properties on the DPM server.

 

From an elevated command prompt:

 

netsh int tcp set global congestion=none

netsh int tcp set global rss=disabled

netsh int tcp set global chimney=disabled

 

 

Another thing to try, re-run a job that will fail. Once it fails grab the MSDPMCurr.errlog and DPMRACurr.errlog from C:\Program Files\Microsoft System Center 2012\DPM\DPM\Temp and chuck them up here so I can see if I can spot anything.

[Marshall_IT]

Thanks, I'll try that tomorrow!

[Marshall_IT]

i'm using SCDPM 2012 R2

 

I have attached the logs.

 

I have been able to get some of the files/folder replicas to become OK again but they eventually revert to inconsistent and then fail consistency checks again.

 

Some other details i'm getting seem to be related to VSS errors.

 

An unepected error occurred on DPM server machine running a VSS operation. (ID 207 Details: VssError:The shadow copy provider had an error. Check the System and Application event Logs for more information (0x80042306)

 

And the event logs show and access denied error.

DPMRACurr.zip

Edited February 12, 2014 by Marshall_IT

[adamf]

Have you installed UR1 for DPM 2012 R2? Issues that are fixed in System Center 2012 R2 Data Protection Manager Update Rollup 1

 

There are a lot of 0x80070005 errors in your logs. I had an issue with the consistency of 2 file shares the remained inconsistent from when I upgraded to R2, ended up opening a case with Microsoft for which I got a private fix for - the issue I was seeing is fixed in UR1.

[win]

Have you tried upgrading or re-installing the DPM agents?

[the_dude]

I was having this issue a lot and what solved it for me was to upgrade my dpm server to faster hardware, and attached storage vs. iSCSI. It still happens every once in a while, but it is rare.

[Marshall_IT]

It might be the iSCSI rather than direct attached storage although this did occur with my old hardware.

 

Thanks for the suggestions guys.

 

I've updated windows, dpm and all the agents now.

 

Still looking for solutions.

[win]

Ahhh, i remember how i got rid of that problem now. I recall it happening quite a long time ago and i had to do something risky but it worked. I deleted the replicas on the HDD and they were created again from scratch.

 

When you upgraded the DPM did you format the dynamic disk as well? This will destroy all the local backups but should recreate them with no errors.

[Marshall_IT]

It's a relatively new installation. I could delete all the replicas as I have copies in backup exec to at the moment.

[win]

As you have backups already, just to be sure they are deleted you may as well format the dynamic HDD that it stores the local backups on. Then run a consistency check during a quiet period and let it synchronise. Hopefully that should sort your problem.

[win]

Not sure if this was solved, but i just had the same problem with a new agent install on a domain controller. I backed up system state only and replica failed. I then added bare metal, and this also failed. Then i removed bare metal, and it worked! Weird, but this is MS software so what else do you expect :/