DPM 2012 Keeps Crashing - Backup to Tape Job

[MYK-IT]

DPM 2012 SP1 Installed on Windows 2008 R2 Storage Server w/TL2000 LTO4 Tape Library

 

We've had DPM 2012 SP1 running fine for a long time now, but just this week we've been having an issue with a particular backup to tape job.

 

The job failed, and despite attempting to re-run the job again after running for 20 minutes or so, copying GBs of data the DPM Console Crashes

 

Affected area: -

Occurred since: 10/10/2013 13:26:18

Description: The back up to tape job failed for the following reason: (ID 3311)

More information

Detailed Errors View Detailed Errors

Recommended action: Run backup to tape job again...

Resolution: To dismiss the alert, click below

Inactivate

 

The DPM service terminated unexpectedly during completion of the job. The termination may have been caused by a system reboot. (ID 910)

 

The DPM Console (on the client) closes relating to the fact that it has no connection to the DPM 2012 Server and/or SQL Agents etc.

 

On the DPM 2012 Server Event Log:

 

Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {05302500-b563-11d7-8128-00b0d0a16cd7} and Name HWPRV is [0x80040154, Class not registered

].

 

The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

 

The DPM 2012 "click for more information" opens to the usual blank place-holder on Microsoft's website either stating the same error information, or not information at all.

 

The DPM 2012 Server is up-to-date and has also been cold rebooted, but problem still occurs. SQL database problem perhaps?

 

Any help would be appreciated.

[MYK-IT]

Sorry to this already, but I'm still experiencing errors on backup to tape (long term) sessions, with these errors occurring on more (but not all) protected members.

 

Again Microsoft's online help for DPM2012 is shocking, very sparse or limited information besides repeating the content of the error message!

[adamf]

Have you had a look in the TechNet forums? They are pretty helpful. Although, speaking from experience, when it comes to console/service crashes and such, you are better off opening a case with Microsoft so they can diagnose the crash. We've had to do this a couple of times.

[MYK-IT]

Hi, yes spent past few days looking at Technet!

 

Some more progress, I have isolated the issue to one particular media tape in the tape library.

 

I think the backup jobs were (re)attempting to back up to this media tape first, and something is going wrong.

 

Now, I could (attempt) to remove the (problematic?) media tape, on the assumption that it is damaged etc., but DPM Status states "Tape available" but with no option to "mark tape as free". And despite having no data it is still associated with a Protection Group.

 

If I choose to erase the media tape, it attempts to, then the errors (listed above) happen again loosing initial connection to DPM server etc.

I have also tried to use ForceFree-Tape.ps1 to mark the tape as free, and it does show the various datasets being removed from the Protection Group. But then PowerShell and DPM 2012 crashes at the last stage whilst attempting to mark this particular media tape as free.

 

I am starting to think that the media tape problem is just a symptom, but the cause is related to (corrupt?) SQL database? Not my strongest area.

 

Any suggestions? As I don’t really want to just stop using this one media tape in case something more fundamental is the problem.

 

Thanks,

Edited October 15, 2013 by MYK-IT

[adamf]

The last problem we had with DPM and it crashing/not backing up to tape was due to media being stuck in the drive from an SQL point of view. The engineer at Microsoft wrote a few custom SQL scripts to resolve the error in the database.

 

Removing the suspect media from rotation is probably a good starting point and see if the error reoccurs, but if it does, I suspect your best shot would be to open a support case.

[MYK-IT]

Yes, looking like a SQL DB Corruption of some form...

 

After attempting to erase (or using ForceFree-Tape.ps1) on one particular media tape this is the first error logged in Event Viewer.

Log Name: Application

Source: MSDPM

Date: 16/10/2013 10:16:34

Event ID: 943

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: DPMSERVER.DOMAIN.local

Description:

The description for Event ID 943 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

 

If the event originated on another computer, the display information had to be saved with the event.

 

The following information was included with the event:

 

Unable to connect to the DPM database because the database is in an inconsistent state.

 

Problem Details:

<__System>1929510/16/2013 9:16:34 AMDpmThreadPool.cs163TrueDBCorruptionExceptionDLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext.Close(Boolean disallowReuseConnection)

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext.Close()

at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.Replica.DbContext.Close()

at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.MMCatalog.MoveMediaToNonAdminPool(Guid mediaId, MediaPoolType mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.MediaManager.MoveToNonAdminPool(Guid mediaId, UInt32 mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CMediaManagerServices.MoveToNonAdminPool(_GUID mediaId, UInt32 mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.MoveToNonAdminPool(CCoreServices* , _GUID mediaId, UInt32 mediaPoolType, tagSAFEARRAY** exceptionResult)

 

 

the message resource is present but the message is not found in the string/message table

 

Event Xml:

943

2

0

0x80000000000000

148964

Application

DPMSERVER.DOMAIN.local

Unable to connect to the DPM database because the database is in an inconsistent state.

 

Problem Details:

<__System>1929510/16/2013 9:16:34 AMDpmThreadPool.cs163TrueDBCorruptionExceptionDLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001Microsoft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRetryCommand.ValidateReturnValue(String errorTitle)

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext.Close(Boolean disallowReuseConnection)

at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlContext.Close()

at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalog.Replica.DbContext.Close()

at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabase.MMCatalog.MoveMediaToNonAdminPool(Guid mediaId, MediaPoolType mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.MMInterface.MediaManager.MoveToNonAdminPool(Guid mediaId, UInt32 mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CMediaManagerServices.MoveToNonAdminPool(_GUID mediaId, UInt32 mediaPoolType)

at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CCoreServices.MoveToNonAdminPool(CCoreServices* , _GUID mediaId, UInt32 mediaPoolType, tagSAFEARRAY** exceptionResult)

ent>

Edited October 16, 2013 by MYK-IT

[adamf]

You could try running dbcc checkdb in the SQL management studio against DPMDB which might give you some idea as to the extent of the corruption. Do you have a backup of your DPMDB on a secondary DPM server or a standard SQL backup of the db?

[MYK-IT]

Hi @adamf

 

Yes, I ran the dbcc checkdb command earlier today and it completed ok, without any errors.

 

DPM 2012 still continues to work fine, as in Disk>Disk backups and restores; the fault only occurs when I attempt to use/backup to/erase/'mark as free' one particular media tape (or even using the ForceFree-Tape.ps1 powershell script on it). I even modified the protection group this problematic media tape relates to, by removing the long-term backup and trying again. But the fault still happens.

 

This "issue" may have been hidden for ages, and has only come about since recently loading the tape library with expired media tapes (to re-use). And I am thinking that if I never put this particular media tape in I wouldn't have the issue I have now. But all the same, there is something still not right.

 

Besides restoring the DPM SQL DB from an earlier date (but as mentioned it'd be difficult to decide how far to go back) I am going to try the following:

 

After backing up DB first!, I will stop using the problematic media tape and manually remove it by these method(s):

Remove lost or broken Tapes out of the DPM DB / Tape Report DPM2012 SP1 RU3

How to Add or Remove a Tape

and try again.

 

I am still surprised by the lack of incomplete online MS Help related to DPM 2012, consdering it has been out 18 months, and SP1 & Update 3 has been release too within that time. And SC 2012 R2 too!

[edanto]

Hi MYK-IT. I know this is a very old thread, but I'm curious to know how your fix worked because I'm seeing the same symptoms.