VEEAM - Error: Failed to perform incremental file version check

[Koldov]

I decided just before the holidays to drag myself (kicking and screaming mind you) into the 21st and set-up Veeam.

 

Previously I'd been using WSB and although it was usually fairly robust, it had a couple of little quirks (one of which was that it occasionally filled the storage and wiped all previous restore points)... Anyway, as everybody here seems to rave about Veeam I thought I'd put it on the main server and have a rock solid, reliable back-up solution...

 

It worked fine over the holidays and dutifully emailed me everyday to tell me it had completed, but since then it's started giving me errors and not completing the back-up.

 

Now, part of the problem may be that we use it in a slightly unconventional way (I say that, but I mean 'legacy') as these are backed up onto rotating removable USB disks for off-site storage, however I researched this aspect and found that it was something that Veeam was able to do.

 

We have the following backup jobs set:

 

1. Server OS - Windows Agent Backup

2. VM1 - Hyper-V Backup

3. VM2 - Hyper-V Backup

4. File Share - File Backup

 

So, the main issue is that since returning from the holidays I have swapped the disks (obviously the previous disk had been attached for just over a week) and one of last nights backup jobs (File Share - File Backup) failed with the following error:

 

"Error: Failed to perform incremental file version check"

 

I'm not sure if leaving the previous disk attached for so long has meant that Veeam is unable to process the 320GB of files to check for incremental changes, but I'd presumed it was up to that task and also there wouldn't have been many (if any) file changes in the staff share over the holiday.

 

I've put that error into Google and come back with zero hits, so either I'm the first person to ever have this issue or everybody else managed to sort it out without asking the internet... I've done a 'Health Check' and it didn't report any errors, but this is my first big server back-up experience with Veeam, so I don't really know what I'm doing...

 

I've manually retried the job multiple times this morning but had some strange and varied results (all ultimately failing).

 

03/01/2024 19:36:57 :: Backed up 225 files and 144 folders (515 MB) - 03/01/2024 19:37:00 : Error

0 of 225 files processed : Job finished with error at 03/01/2024 19:36:59

 

04/01/2024 07:44:58 :: Backed up 0 files and 41144 folders (0 B)

 

04/01/2024 08:02:55 :: Backed up 2 files and 2 folders (330 B)

 

04/01/2024 08:09:24 :: Backed up 0 files and 2 folders (0 B)

[pete]

My guess is that you're doing incremental backups and Veeam expected to be able to refer to files on the (now removed) USB disk.

 

Veeam (by default, you can choose something else) uses forever-forward incrementals. Say you have a 60 day backup cycle for something.

 

1st Backup is a full

2nd is incremental

3rd is Incremental

...

60th is incremental.

61st also incremental but merges the second backup into the first (full) backup.

 

And it verifies that everythings there as part of the backup process so you have a working, restorable backup. If it can't find a file in the chain it will tell you.

 

Look into the "create active full backups periodically" setting or have a second job that puts fulls on removable storage.

[Koldov]

My guess is that you're doing incremental backups and Veeam expected to be able to refer to files on the (now removed) USB disk.

 

Hmmm... obviously possible and does make sense. I can't say otherwise, but from my reading the point of choosing 'this backup is on rotated disks' was that Veeam could handle the rotation of the disks?

 

https://helpcenter.veeam.com/docs/backup/vsphere/rotated_drives_hiw.html?ver=120

 

Maybe someone can put that into plain English as it doesn't really sink into my old brain...?

 

Maybe there's a configuration error, but I was expecting from reading that, it would create a full backup on one disk then increment until removal. Then, create a full backup on the second disk and increment until removal.

 

When the first disk is reattached it would have to process what has changed since it was removed and back that up (it wouldn't need to know what is on the second disk, just what's changed since its own last incremental).

[LeMarchand]

Probably not the "official" way, but I have two separate backup jobs: "Disk 1" and "Disk 2", which still needs some finessing but seems to work

[PotNoodleTech]

This is beyond my area of expertise as I have only backup up with Veeam to fixed and constantly available drives (server, nas, san, cloud) as opposed to rotating USB drives, but I am sure I remember you have to set up the USB drives in a certain way (like a pool of drives that can be used) rather than just plugging in a random drive and hoping it works?

 

For most methods Veeam constantly refers to previously held data (old backups) to do incremental backups and file scrubbing of old data, so if you have then removed the old backup drive(s) I can see why you are getting lots of errors as Veeam won't be expecting to not be able to see that data again.

Edited January 4, 2024 by PotNoodleTech

[Koldov]

Probably not the "official" way, but I have two separate backup jobs: "Disk 1" and "Disk 2", which still needs some finessing but seems to work

 

How do you tell Veeam which disk is attached, so it knows which job to run (Disk 1 or Disk 2), or do you have a fixed schedule (Monday runs Disk 1 and you have to have Disk 1 attached, Tuesday runs Disk 2 and you have to have Disk 2 attached)?

 

I'm all for making things simple and that sounds like a plan, I even thought about having one permanently attached and then just attach the other in the morning (or whenever) and copy the disk contents to the new one (unless there is an 'automatic way to copy the backup off) and then remove it for off-site.

 

This is beyond my area of expertise as I have only backup up with Veeam to fixed and constantly available drives (server, nas, san, cloud) as opposed to rotating USB drives, but I am sure I remember you have to set up the USB drives in a certain way (like a pool of drives that can be used) rather than just plugging in a random drive and hoping it works?

 

For most methods Veeam constantly refers to previously held data (old backups) to do incremental backups and file scrubbing of old data, so if you have then removed the old backup drive(s) I can see why you are getting lots of errors as Veeam won't be expecting to not be able to see that data again.

 

Sorry, I thought I'd addressed that in my previous post in answer to @pete - Veeam has a setting where you can specify the backup is on 'rotated disks' and from reading the article in the link, I presumed it would create a full on each disk and then increment whatever was not on the full. As in, it would not refer to Disk 1 when Disk 2 was attached, it would only need to know what had changed since the last time it (Disk 2) was attached and what it needs to backup, then when Disk 1 was attached it would not need to refer to Disk 2, only what had changed since it (Disk 1) was attached. I do get your point though that there must be some way for it to know what was on the last backup and for it to process what has changed, but I thought that information would be stored on each individual disk.

 

Both disks are listed in the storage repositories

[Sephiroth]

The way to do this is to tell Veeam that your repository (the USB drive location) that it is backed by rotated drives.

 

If you go into Backup infrastructure > Backup Repositories > Your rotated disks > Properties > Repository > Advanced - you should see this option.

 

----

Edit: I didn't read the last post properly, What I've said, and what you say you've got should work for you. With that setting, Veeam should treat a missing restore point as a newly rotated disk and create a full backup, rather than try to increment it.

Edited January 4, 2024 by Sephiroth

[Koldov]

It must be something to do with the backup job as there must be something in the settings or config that's confusing it, or even the type of job.

 

The disk also contains another partition for the OS job and the 2x Hyper-V jobs and all of those completed with no issues.

 

They are obviously different types of job so maybe not directly related as the options for each job are different, but it's number 4 that's failing..

 

1). Server OS - Windows Agent Backup = Volume Level Backup (OS volume) - Retention policy: 1 days (I think this is incremental - there is no way to set anything else so it is at the default, whatever that may be).

 

2 & 3). 2x VM - Hyper-V Backup = Reverse Incremental - Retention policy: 1 days

 

4). FILE SHARE - File Backup = Files and Folders - File Version Limit 1 day (I think this is incremental - there is no way to set anything else so it is at the default, whatever that may be).

[Steve21]

I appreciate this is kinda bypassing your actual post, but is there a reason you’re doing a file share backup at all?

 

That’s not really designed for a Veeam as you can restore files directly via the VM backup so there little purpose having a file share backup (that’s more for external NAS etc that isn’t structured files)

 

File share backups are only forever incremental and work differently with CRC which I’m guessing may be your issue

 

Steve

[Koldov]

Yes, thanks.... unfortunately this is where the off topic 'my god man, why are you doing it like that' posts start... because now I have to admit that the file share is a legacy monster I haven't got round to dealing with yet and it is sitting on a 'drive' on the actual physical server and isn't a VM.

 

I know most people only see Veeam as a VM backup solution, but I thought it did everything... maybe I should just go back to 2008 and use WSB for the file share? At least I know what I'm doing with that and it's partly what it was designed for.

Edited January 4, 2024 by Koldov

[Steve21]

Haha sorry

 

Loosely that isn’t a problem, you can add the Veeam Agent to a physical server to add it as a backup target to Veeam without it being a VM to give you the other features you’re missing from doing a file share backup

 

It’s just the technology is different, file shares don’t do full/incremental like that to be able to rotate (at least not that I’ve seen previously)

 

It’s always a incremental via CRC check so if it can’t find the CRC file I assume that’s where it’s complaining

 

Might be worth trying that way to see if that helps with the rotation as shouldn’t need much changing with the agent instead

 

Steve

[LeMarchand]

OK, probably a better way to do this, but we have 4 "daily" disks and 2 "weekend replication" ones. The job structure looks like this.

 

[ATTACH=CONFIG]70582[/ATTACH]

 

The weekday ones are fairly simple; I just eject the disk and disable/enable jobs. The weekend replications are a bit more of a faff.

 

That said,I think @Steve21 has identified the source of the problem.

 

VEEAM should be able to schedule copy jobs, btw.