+ Post New Thread
Results 1 to 8 of 8
Enterprise Software Thread, DPM 2012 Keeps Crashing - Backup to Tape Job in Technical; DPM 2012 SP1 Installed on Windows 2008 R2 Storage Server w/TL2000 LTO4 Tape Library We've had DPM 2012 SP1 running ...
  1. #1

    Join Date
    Dec 2007
    Posts
    847
    Thank Post
    86
    Thanked 160 Times in 135 Posts
    Rep Power
    47

    DPM 2012 Keeps Crashing - Backup to Tape Job

    DPM 2012 SP1 Installed on Windows 2008 R2 Storage Server w/TL2000 LTO4 Tape Library

    We've had DPM 2012 SP1 running fine for a long time now, but just this week we've been having an issue with a particular backup to tape job.

    The job failed, and despite attempting to re-run the job again after running for 20 minutes or so, copying GBs of data the DPM Console Crashes

    Affected area: -
    Occurred since: 10/10/2013 13:26:18
    Description: The back up to tape job failed for the following reason: (ID 3311)
    More information
    Detailed Errors View Detailed Errors
    Recommended action: Run backup to tape job again...
    Resolution: To dismiss the alert, click below
    Inactivate
    The DPM service terminated unexpectedly during completion of the job. The termination may have been caused by a system reboot. (ID 910)
    The DPM Console (on the client) closes relating to the fact that it has no connection to the DPM 2012 Server and/or SQL Agents etc.

    On the DPM 2012 Server Event Log:

    Volume Shadow Copy Service error: A critical component required by the Volume Shadow Copy service is not registered. This might happened if an error occurred during Windows setup or during installation of a Shadow Copy provider. The error returned from CoCreateInstance on class with CLSID {05302500-b563-11d7-8128-00b0d0a16cd7} and Name HWPRV is [0x80040154, Class not registered
    ].
    The description for Event ID 999 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
    The DPM 2012 "click for more information" opens to the usual blank place-holder on Microsoft's website either stating the same error information, or not information at all.

    The DPM 2012 Server is up-to-date and has also been cold rebooted, but problem still occurs. SQL database problem perhaps?

    Any help would be appreciated.

  2. #2

    Join Date
    Dec 2007
    Posts
    847
    Thank Post
    86
    Thanked 160 Times in 135 Posts
    Rep Power
    47
    Sorry to this already, but I'm still experiencing errors on backup to tape (long term) sessions, with these errors occurring on more (but not all) protected members.

    Again Microsoft's online help for DPM2012 is shocking, very sparse or limited information besides repeating the content of the error message!

  3. #3

    Join Date
    Apr 2007
    Location
    Croydon
    Posts
    493
    Thank Post
    18
    Thanked 30 Times in 29 Posts
    Rep Power
    20
    Have you had a look in the TechNet forums? They are pretty helpful. Although, speaking from experience, when it comes to console/service crashes and such, you are better off opening a case with Microsoft so they can diagnose the crash. We've had to do this a couple of times.

  4. #4

    Join Date
    Dec 2007
    Posts
    847
    Thank Post
    86
    Thanked 160 Times in 135 Posts
    Rep Power
    47
    Hi, yes spent past few days looking at Technet!

    Some more progress, I have isolated the issue to one particular media tape in the tape library.

    I think the backup jobs were (re)attempting to back up to this media tape first, and something is going wrong.

    Now, I could (attempt) to remove the (problematic?) media tape, on the assumption that it is damaged etc., but DPM Status states "Tape available" but with no option to "mark tape as free". And despite having no data it is still associated with a Protection Group.

    If I choose to erase the media tape, it attempts to, then the errors (listed above) happen again loosing initial connection to DPM server etc.
    I have also tried to use ForceFree-Tape.ps1 to mark the tape as free, and it does show the various datasets being removed from the Protection Group. But then PowerShell and DPM 2012 crashes at the last stage whilst attempting to mark this particular media tape as free.

    I am starting to think that the media tape problem is just a symptom, but the cause is related to (corrupt?) SQL database? Not my strongest area.

    Any suggestions? As I don’t really want to just stop using this one media tape in case something more fundamental is the problem.

    Thanks,
    Last edited by MYK-IT; 15th October 2013 at 03:23 PM.

  5. #5

    Join Date
    Apr 2007
    Location
    Croydon
    Posts
    493
    Thank Post
    18
    Thanked 30 Times in 29 Posts
    Rep Power
    20
    The last problem we had with DPM and it crashing/not backing up to tape was due to media being stuck in the drive from an SQL point of view. The engineer at Microsoft wrote a few custom SQL scripts to resolve the error in the database.

    Removing the suspect media from rotation is probably a good starting point and see if the error reoccurs, but if it does, I suspect your best shot would be to open a support case.

  6. #6

    Join Date
    Dec 2007
    Posts
    847
    Thank Post
    86
    Thanked 160 Times in 135 Posts
    Rep Power
    47
    Yes, looking like a SQL DB Corruption of some form...

    After attempting to erase (or using ForceFree-Tape.ps1) on one particular media tape this is the first error logged in Event Viewer.
    Log Name: Application
    Source: MSDPM
    Date: 16/10/2013 10:16:34
    Event ID: 943
    Task Category: None
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: DPMSERVER.DOMAIN.local
    Description:
    The description for Event ID 943 from source MSDPM cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

    If the event originated on another computer, the display information had to be saved with the event.

    The following information was included with the event:

    Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:
    <FatalServiceError><__System><ID>19</ID><Seq>295</Seq><TimeCreated>10/16/2013 9:16:34 AM</TimeCreated><Source>DpmThreadPool.cs</Source><Line>163</Line><HasError>True</HasError></__System><ExceptionType>DBCorruptionException</ExceptionType><ExceptionMessage>DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001</ExceptionMessage><ExceptionDetails>Microsoft.Inter nal.EnterpriseStorage.Dls.DB.DBCorruptionException : DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRet ryCommand.ValidateReturnValue(String errorTitle)
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlCon text.Close(Boolean disallowReuseConnection)
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlCon text.Close()
    at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalo g.Replica.DbContext.Close()
    at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabas e.MMCatalog.MoveMediaToNonAdminPool(Guid mediaId, MediaPoolType mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.MMInterfa ce.MediaManager.MoveToNonAdminPool(Guid mediaId, UInt32 mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CM ediaManagerServices.MoveToNonAdminPool(_GUID mediaId, UInt32 mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CC oreServices.MoveToNonAdminPool(CCoreServices* , _GUID mediaId, UInt32 mediaPoolType, tagSAFEARRAY** exceptionResult)</ExceptionDetails></FatalServiceError>


    the message resource is present but the message is not found in the string/message table

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
    <Provider Name="MSDPM" />
    <EventID Qualifiers="0">943</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-10-16T09:16:34.000000000Z" />
    <EventRecordID>148964</EventRecordID>
    <Channel>Application</Channel>
    <Computer>DPMSERVER.DOMAIN.local</Computer>
    <Security />
    </System>
    <EventData>
    <Data>Unable to connect to the DPM database because the database is in an inconsistent state.

    Problem Details:
    &lt;FatalServiceError&gt;&lt;__System&gt;&lt;ID&gt ;19&lt;/ID&gt;&lt;Seq&gt;295&lt;/Seq&gt;&lt;TimeCreated&gt;10/16/2013 9:16:34 AM&lt;/TimeCreated&gt;&lt;Source&gt;DpmThreadPool.cs&lt;/Source&gt;&lt;Line&gt;163&lt;/Line&gt;&lt;HasError&gt;True&lt;/HasError&gt;&lt;/__System&gt;&lt;ExceptionType&gt;DBCorruptionExcep tion&lt;/ExceptionType&gt;&lt;ExceptionMessage&gt;DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001&lt;/ExceptionMessage&gt;&lt;ExceptionDetails&gt;Micros oft.Internal.EnterpriseStorage.Dls.DB.DBCorruptionException: DLS error: SP='dbo.prc_MM_MediaPool_GetOnMediaPoolId' Error=50001
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlRet ryCommand.ValidateReturnValue(String errorTitle)
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlCon text.Close(Boolean disallowReuseConnection)
    at Microsoft.Internal.EnterpriseStorage.Dls.DB.SqlCon text.Close()
    at Microsoft.Internal.EnterpriseStorage.Dls.PRMCatalo g.Replica.DbContext.Close()
    at Microsoft.Internal.EnterpriseStorage.Dls.MMDatabas e.MMCatalog.MoveMediaToNonAdminPool(Guid mediaId, MediaPoolType mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.MMInterfa ce.MediaManager.MoveToNonAdminPool(Guid mediaId, UInt32 mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CM ediaManagerServices.MoveToNonAdminPool(_GUID mediaId, UInt32 mediaPoolType)
    at Microsoft.Internal.EnterpriseStorage.Dls.Engine.CC oreServices.MoveToNonAdminPool(CCoreServices* , _GUID mediaId, UInt32 mediaPoolType, tagSAFEARRAY** exceptionResult)&lt;/ExceptionDetails&gt;&lt;/FatalServiceError&gt;
    </Data>
    ent>
    Last edited by MYK-IT; 16th October 2013 at 10:22 AM.

  7. #7

    Join Date
    Apr 2007
    Location
    Croydon
    Posts
    493
    Thank Post
    18
    Thanked 30 Times in 29 Posts
    Rep Power
    20
    You could try running dbcc checkdb in the SQL management studio against DPMDB which might give you some idea as to the extent of the corruption. Do you have a backup of your DPMDB on a secondary DPM server or a standard SQL backup of the db?

  8. #8

    Join Date
    Dec 2007
    Posts
    847
    Thank Post
    86
    Thanked 160 Times in 135 Posts
    Rep Power
    47
    Hi @adamf

    Yes, I ran the dbcc checkdb command earlier today and it completed ok, without any errors.

    DPM 2012 still continues to work fine, as in Disk>Disk backups and restores; the fault only occurs when I attempt to use/backup to/erase/'mark as free' one particular media tape (or even using the ForceFree-Tape.ps1 powershell script on it). I even modified the protection group this problematic media tape relates to, by removing the long-term backup and trying again. But the fault still happens.

    This "issue" may have been hidden for ages, and has only come about since recently loading the tape library with expired media tapes (to re-use). And I am thinking that if I never put this particular media tape in I wouldn't have the issue I have now. But all the same, there is something still not right.

    Besides restoring the DPM SQL DB from an earlier date (but as mentioned it'd be difficult to decide how far to go back) I am going to try the following:

    After backing up DB first!, I will stop using the problematic media tape and manually remove it by these method(s):
    Remove lost or broken Tapes out of the DPM DB / Tape Report DPM2012 SP1 RU3
    How to Add or Remove a Tape
    and try again.

    I am still surprised by the lack of incomplete online MS Help related to DPM 2012, consdering it has been out 18 months, and SP1 & Update 3 has been release too within that time. And SC 2012 R2 too!

SHARE:
+ Post New Thread

Similar Threads

  1. Backup to DPM 2012 Long Term
    By rpycroft in forum Windows Server 2008 R2
    Replies: 1
    Last Post: 13th December 2012, 02:09 PM
  2. Backup to Tape - recommendations
    By glennda in forum Enterprise Software
    Replies: 13
    Last Post: 2nd December 2012, 09:42 PM
  3. Disk-to-Disk-to-Tape Backup
    By enjay in forum Hardware
    Replies: 30
    Last Post: 23rd November 2007, 03:21 PM
  4. RM Disk to Disk to Tape Backup Solution
    By Chris in forum General Chat
    Replies: 0
    Last Post: 2nd July 2007, 10:14 AM
  5. Disk to Disk to Tape backup. How do you do it?
    By trekmad in forum How do you do....it?
    Replies: 0
    Last Post: 30th May 2007, 07:49 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •