Symantec Backup Exec 10d for Windows Servers

[plock]

Morning all,

I have a very odd problem that I’ve not seen before. The issue is now reoccurring; it started on Monday night’s backup job.

I receive the following failure notice on Backup Exec:

AOFO: Initialization failure on: "\\SERVERNAME\Shadow?Copy?Components". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
Snapshot provider error (0x800706BF): 0x800706bf (2147944127)
Check the Windows Event Viewer for details.
The following volumes are dependent on resource: "C:" .
Note: Backup Exec did not perform a snapshot of all volumes simultaneously because the option 'Process logical volumes for backup one at a time' was selected as a backup job property.

I have checked this out on the net to find not a great deal! I have also checked the Event Viewer, which reported:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12302
Date: 22/01/2008
Time: 20:04:27
User: N/A
Computer: SERVERNAME
Description: Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers. Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 2d 20 43 6f 64 65 3a 20 - Code:
0008: 42 55 45 43 58 4d 4c 43 BUECXMLC
0010: 30 30 30 30 34 32 31 38 00004218
0018: 2d 20 43 61 6c 6c 3a 20 - Call:
0020: 42 55 45 43 58 4d 4c 43 BUECXMLC
0028: 30 30 30 30 34 31 37 30 00004170
0030: 2d 20 50 49 44 3a 20 20 - PID:
0038: 30 30 30 30 31 33 33 36 00001336
0040: 2d 20 54 49 44 3a 20 20 - TID:
0048: 30 30 31 35 33 34 33 36 00153436
0050: 2d 20 43 4d 44 3a 20 20 - CMD:
0058: 22 43 3a 5c 50 72 6f 67 "C:\Prog
0060: 72 61 6d 20 46 69 6c 65 ram File
0068: 73 5c 56 45 52 49 54 41 s\VERITA
0070: 53 5c 42 61 63 6b 75 70 S\Backup
0078: 20 45 78 65 63 5c 52 41 Exec\RA
0080: 4e 54 5c 62 65 72 65 6d NT\berem
0088: 6f 74 65 2e 65 78 65 22 ote.exe"
0090: 2d 20 55 73 65 72 3a 20 - User:
0098: 4a 4d 41 5c 62 65 73 65 USERNAME
00a0: 72 76 69 63 65 20 20 20
00a8: 2d 20 53 69 64 3a 20 20 - Sid:
00b0: 53 2d 31 2d 35 2d 32 31 S-1-5-21
00b8: 2d 34 38 39 38 32 38 38 -4898288
00c0: 38 35 2d 34 32 35 35 35 85-42555
00c8: 36 30 38 36 39 2d 31 31 60869-11
00d0: 31 37 34 37 39 30 39 30 17479090
00d8: 2d 33 31 39 34 20 20 20 -3194

Along with the next immediate entry:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12302
Date: 22/01/2008
Time: 20:04:25
User: N/A
Computer: SERVERNAME
Description: Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers. Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 2d 20 43 6f 64 65 3a 20 - Code:
0008: 42 55 45 43 58 4d 4c 43 BUECXMLC
0010: 30 30 30 30 34 32 31 38 00004218
0018: 2d 20 43 61 6c 6c 3a 20 - Call:
0020: 42 55 45 43 58 4d 4c 43 BUECXMLC
0028: 30 30 30 30 34 31 37 30 00004170
0030: 2d 20 50 49 44 3a 20 20 - PID:
0038: 30 30 30 30 31 33 33 36 00001336
0040: 2d 20 54 49 44 3a 20 20 - TID:
0048: 30 30 31 35 33 34 34 38 00153448
0050: 2d 20 43 4d 44 3a 20 20 - CMD:
0058: 22 43 3a 5c 50 72 6f 67 "C:\Prog
0060: 72 61 6d 20 46 69 6c 65 ram File
0068: 73 5c 56 45 52 49 54 41 s\VERITA
0070: 53 5c 42 61 63 6b 75 70 S\Backup
0078: 20 45 78 65 63 5c 52 41 Exec\RA
0080: 4e 54 5c 62 65 72 65 6d NT\berem
0088: 6f 74 65 2e 65 78 65 22 ote.exe"
0090: 2d 20 55 73 65 72 3a 20 - User:
0098: 4a 4d 41 5c 62 65 73 65 USERNAME
00a0: 72 76 69 63 65 20 20 20
00a8: 2d 20 53 69 64 3a 20 20 - Sid:
00b0: 53 2d 31 2d 35 2d 32 31 S-1-5-21
00b8: 2d 34 38 39 38 32 38 38 -4898288
00c0: 38 35 2d 34 32 35 35 35 85-42555
00c8: 36 30 38 36 39 2d 31 31 60869-11
00d0: 31 37 34 37 39 30 39 30 17479090
00d8: 2d 33 31 39 34 20 20 20 -3194

I've gone through the following site I come across, the only site relevant to the Event ID. http://www.itnewsgroups.net/group/mi...topic6159.aspx

I don't have any errors when looking at the backup logs though, nor do I receive errors when running 'Vssadmin list writers' in CMD.

I've had a look at the VSS updates suggested but the symptoms don't match. The symptoms don't show my Event ID at all?

Has anyone else had this problem? Any help greatly appreciated!

[Heebeejeebee]

We have the very same problem! I never got to the bottom of it either. It happened one day after some Windows Updates were applied but I'm not sure if that's what caused it. As far as I can remember I came across a page on the support site that said it's an MS issue and it'll be fixed in a subsequent service pack (or patch) but seems to have never made it that far.

Watching with interest......

HBJB

[plock]

Interesting - now you mention it I did read that on the support site.

Am I right in thinking this is causing the server to fail on the backup entirely? Luckily it's only our second domain controller, I can always rebuild this relatively quickly if a disaster was to occur. We also have an old backup previous to this error.

Nightmare!

[localzuk]

Same here, and same conditions as Heebeejeebee. Seems to have been after some form of update.

[ashok]

Hi,

How you tried to do a live update for symantec, it may be that they may have fix for issues like this. Is BackupExec 10d up to date with the SP and hotfixes?

Ash.

[localzuk]

Originally Posted by ashok

Hi,

How you tried to do a live update for symantec, it may be that they may have fix for issues like this. Is BackupExec 10d up to date with the SP and hotfixes?

Ash.

The problem also exists in 11d - which is fully up to date on my server.

[cookie_monster]

Have you applied last months KB943485 LSASS update and KB 941644 IP update?

I'm running 10D on 2003 R2 and haven't seen this issue yet but i haven't installed those updates yet. We have installed SP2 though.

As mentioned above make sure you have the latest patches applied to BE.

Oddly MS claim to of fixed a similar issue in SP1.

[plock]

I'm currently running revision 5629. I'm about to apply this hotfix which isn't applied as yet: http://seer.entsupport.symantec.com/docs/289185.htm

I'll look to see if those KB's were installed. I personally haven’t installed any updates on the server for a while. Most definitely not in the last three months. I believe I've set it up so it downloads them and awaits my confirmation before installation also.

Right - Looks although I hadn't set it up as I thought. Both thoses KB's were installed on the 9th Jan. My backup job then started failing on the 21st. That doesn't suggest it was those KB's?

[cookie_monster]

In that case it seems unlikely to be those patches but it's not impossible. See what results you get from the Symantec patches.

[Heebeejeebee]

Originally Posted by localzuk

The problem also exists in 11d - which is fully up to date on my server.

I forgot to mention we're on 11d as well.

If I untick Shadow Copy Components all is well with the backup job.

HBJB

[plock]

Applied the above patch and all is well again. Very odd!

[ashok]

Hi all,

We got this issue today on our admin server which is running 10d with SQL agent. One of my guys called symantec and they said that this is a MS issue with Shadow copy service and not symantec issue and that MS have a patch which will fix this issue.

The patch to apply is KB940349 - it requires a reboot of the server. We have applied the patch today and will restart the server at the end of school today to see what happens.

Ash.

[plock]

@ ashok - how did it go?

Whatever next, I have a new VSS error on a different server now! Here is the Backup Exec error message:

AOFO: Initialization failure on: "Shadow?Copy?Components". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
VSS Snapshot error. Microsoft Volume Shadow Copy Service (VSS) cannot communicate with the writers. Check that the Volume Shadow Copy Service is started, and check the Windows Event Viewer for errors related to this service.
Note: Backup Exec did not perform a snapshot of all volumes simultaneously because the option 'Process logical volumes for backup one at a time' was selected as a backup job property.

Anyone have this message? :S

[ashok]

@plock,

Yes it worked and out admin server the following day after applying the patch backed up sucessfully and its been working since then.

Seems like the same issues to me but different error message. Have you tried applying the fix on the server? might be worth it as the patch cures lots of problems with VSS writers.

Ash.

[plock]

Cheers 'ashok'!

I'll apply the patch to the problematic server!