Veeam Snapshot Issue

[Gongalong]

Hi folks,

 

I have a backup issue with Veeam.

 

Veeam was installed over the Summer hols, as part of a Hyper-V host replacement, switching from DPM.

 

At the start of the week I tried to restore from the D drive of one of the file servers and the drive wasn't listed in the explorer, even though it was clearly part of the backup based on the size. I contacted Veeam support and they said it was because all Windows servers should have application-aware enabled for their respective backup jobs. So I enabled this, and this caused further problems, with jobs failing for six servers. Three of these were resolved because it was due to a lack of SCSI adapter on the VM. Oddly the problem file server is now fine. The two problem servers left are another file server (2008 R2) and the SIMS server (2012 R2).

 

Veeam have sent the following troubleshooting link which I have worked through KB1855: Hyper-V Guest processing skipped (check guest OS VSS state and integration components version)

 

The diskshadow command at the end of the document highlights that the VSS writer for the SIMS server is offline, although not for the file server. Veeam support have suggested restarting the VMM process on the hosts, and restarting the Shadow Copy service on the servers, but neither of these have resolved it. I have in fact rebooted the hosts, and applied Windows updates. They've also said to reinstall integration tools, but I cannot find a way to do this.

 

Veeam currently shows the following pertinent errors in the backup log:

 

For file server...

 

11/09/2015 19:26:54 :: Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Veeam application-aware processing). Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a non-transient error. If the backup process is retried,

the error is likely to reoccur.

--tr:Failed to verify writers state.

--tr:Failed to perform pre-backup tasks.

11/09/2015 19:28:57 :: Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed.

11/09/2015 19:29:01 :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

The writer experienced a non-transient error. If the backup process is retried,

the error is likely to reoccur.

--tr:Failed to verify writers state.

--tr:Failed to perform pre-backup tasks.)

 

For SIMS...

11/09/2015 21:36:09 :: Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent). Details: Failed to prepare guests for volume snapshot.

 

For the file server there are various VSS related errors in event viewer, but too many to post here.

 

I'm at a bit of a loss as to what else I can try. Veeam support are quite slow to respond, and meanwhile we're not getting any backups of these two critical servers.

 

Any ideas?

[robjduk]

Far from an amazing fix but try ensuring the proxy the job uses is the cream server. Also to restore your files, try selecting restore from other os instead of windows. Slower and filled but should show all your files and drives.

[Gongalong]

Application-Aware oddly did fix the missing D drive issue, but now it's the other file server that's failing to backup.

 

I've disabled Application-Aware in the meantime while I wait for Veeam support.

 

The difficulty is that it's one of those support issues that falls between two providers, Microsoft and Veeam in this case. I suspect something is up with the servers, although these hosts have only just been installed and are 2012 R2.

[Steve21]

For the integration part you need to do it via the HyperV server.

 

If you connect to a server from HyperV manager or failover (not RDS) you get different options:

 

In relation to the fails though, how many retries are you running? as I notice with ours if you're doing lots of servers at once they'll often fail 1/2 times, but then work fine another time without any need to change settings just waiting for the retries to occur and it works fine.

 

Example below, same server once during main backup, and then 2 retries (we have 5 retries set in general):

 

Steve

Edited September 13, 2015 by Steve21

[Gongalong]

Ah yes, well I've definitely got Integration Services installed, but I can't reinstall it. I just get the message it's already there. I've googled it also, and there doesn't seem to be any solution to removing and reinstalling.

 

- - - Updated - - -

 

We're running 3 retries. The jobs are sequenced so as not to collide, and even when I run them manually at times the other jobs don't run they still fail.

[Gongalong]

Having spent some weeks back and forth with Veeam support, the issue is either with the host, the VM, or both, as disk shadows are not working properly. Typically if I reboot the VM I can get one disk shadow, and then successive attempts fail. The Hyper-V Volume Shadow Copy Requestor service on the VM will also fail at this point, and I can't stop/restart it.

 

I'm working with our thirdline support to try and figure out what the problem is, but nothing so far.

[Gongalong]

This turned out to be ESET anti-virus. Both the v5 and v6 ESET ERA endpoints are stopping shadowcopies for some reason on this VM. We're just about to test a bug fix from ESET.