VEEAM - install mistakes...

[Koldov]

As a lot of people on here seem to rave about VEEAM, I thought I'd drag myself into the 21st and see what all the fuss was about (especially as I now have a few VMs)...

 

Obviously it's a lot more complicated and involved than WSB and as a result I think I've probably made a couple of schoolboy errors.

 

1. On the install it seems to be installing a lot of unnecessary stuff for Linux, Mac, and also Azure, AWS, Google cloud services, even though I was generally pressing 'next' and accepting the defaults I don't remember seeing if these were choices or not. Is it OK to go into 'uninstall a program' and get rid of these as I don't see anywhere in the console to remove them (or just leave them)?

 

2. Whilst accepting the defaults it seems to have decided that my D:\ drive was the best place to install certain items (namely the following folders):

 

7230c5c96a3aa8856644b7

ProgramData

VBRCatalog

 

I found out later when trying to create a job that this was also where VEEAM decided it was best to set the default 'Backup Repository' but I wasn't aware what this meant at the time and I didn't see anywhere to change them in the install. Unfortunately the D:\ drive is the DATA drive for the VMs and is a fixed drive on the server and not the actual 'Backup Repository' I wanted which I added in later. Do these files need to be on the 'Backup Repository' disk as I'm wondering that if in a DR scenario if I don't have access to these files restore will fail...?

 

3. After the first back-up job ran I checked the VEEAM console which stated success, but the Windows Server event viewer showed a few errors:

 

Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.

 

Discussed here with no solution:

 

https://forums.veeam.com/servers-workstations-f49/event-vss-8194-error-querying-system-writer-access-denied-t83412.html

 

I am also now getting the following errors (which I wasn't before installing VEEAM):

 

ID: 1000, Severity: Error, Source .NET Runtime

 

Category: Veeam.GCP.PlatformService.WebApi.Framework.Common.Middleware.CUnhandledOperationsMiddleware

EventId: 0

ConnectionId: 0HMNJBBM6GFP0

RequestId: 0HMNJBBM6GFP0:00000002

RequestPath: /api/v1/proxies/configurationBackupProperties

SpanId: |3bbbf5ad-4633af528e259818.

TraceId: 3bbbf5ad-4633af528e259818

ParentId:

 

Resource does not exist. Path: [/api/v1/proxies/configurationBackupProperties]

 

This is discussed here... something about disabling the Google service (so may disappear if I uninstall that - in point 1)...

 

https://www.reddit.com/r/Veeam/comments/t6qand/event_viewer_errors_about_api_call_after_upgrade/

 

 

Microsoft Windows Server has detected that NTLM authentication is presently being used between clients and this server.

 

The Open procedure for service "ISAPISearch" in DLL "C:\WINDOWS\system32\query.dll" failed with error code The specified procedure could not be found.

 

The Open procedure for service "ContentIndex" in DLL "C:\WINDOWS\system32\query.dll" failed with error code The specified procedure could not be found.

 

The Open procedure for service "ContentFilter" in DLL "C:\WINDOWS\System32\query.dll" failed with error code The specified procedure could not be found.

 

This is the first day after install, so I'm not sure if these will be recurring errors or not at the moment....

 

4. Is there any easy way to actually see what has been backed up? The disk now just shows a VEEAM .vbm & .vbk file and the job details in the console just show a nice graph with 'TIME, DATA processed and STATUS'... With WSB I'm used to seeing a GUI with volume letters and details of each individual volume, with the volume name, data transferred and backup type.

Edited January 11, 2023 by Koldov

[howartp]

1) I'd just leave them; they're doing no harm.

 

2b) Navigate to Veeam > Backup Infrastructure > Backup Repositories > Add yourself a new local repository wherever you want it, then edit your job to point to it.

 

4) Navigate to Veeam > Files > Windows > (Veeam Server) > Browse to repository > Double-click a .VBK, it will show you which Machine/VM's it contains, and then which restore points of a selected machine.

 

(If Files is hidden, click the little '>>' icon at bottom of the Home/Inventory/Backup Infrastructure tab to turn on/off more views).

[supportman]

You need to make sure you enable "guest file system indexing" in the job properties to see the whole file system.

 

Then you get a nice browser and virtual machine to find files on restore.

[Koldov]

Navigate to Veeam > Backup Infrastructure > Backup Repositories > Add yourself a new local repository wherever you want it, then edit your job to point to it.

 

I worked out how to add a repository and I created a job which points to it, all of that is OK. What I mean is that I'm wondering if any of those files that were originally installed onto the D:\ drive in the server should actually be on the disk that is actually holding the repository to enable a restore? Because it seems to me that VEEAM installed them there for a reason instead of C:\ (VEEAM was thinking that D:\ was where my repository was going to be - as it decided to create the default one there as well as the folders below).

 

To put it simply, should these folders be on the disk that holds the repository?

 

 

You need to make sure you enable "guest file system indexing" in the job properties to see the whole file system.

 

Then you get a nice browser and virtual machine to find files on restore.

 

Will that show the volumes/partitions? I wasn't actually looking for individual file restores, these aren't really servers that I think I'll need to do that on (VM Host and just a VM Print Server and a VM DC).

 

But I can't even tell whether it's actually done a back-up of all the volumes/partitions for the OS drive just by looking at the console (WSB has an issue where it fails to backup the OEM & Recovery partition as they don't have drive letters (although they might not be important for a full restore I don't want to chance it).

Edited January 11, 2023 by Koldov

[Davit2005]

You can add another repository, copy the backup files across and there is then a way to re-link them which is done through the GUI and really easy, it has been a 6 years since I have used Veeam but it seriously is a good bit of software. Used it since version 6 for 4 or so years and upgraded to version 9 before I left the place. v6 used to be able to do Active Directory object restores only by running the VM in a isolated virtual lab which required the upper enterprise license but they then added the Active Directory without needing to do that but there is so much functionality available if you want to pay for it.

 

As for the jobs what they are backing up, check the job settings, the job history should show you but you could try a restore and see what options it gives you as well as checking the job logs.

 

Veeam works in a different way to WSB, it was sort of designed specifically then other options such as backing up physical servers came later rather than WSB which was designed a long time back for physical boxes and sort of adapted a bit for virtual environment.

Edited January 11, 2023 by Davit2005

[Koldov]

You can add another repository, copy the backup files across and there is then a way to re-link them which is done through the GUI and really easy

 

There was no backup done that needs moving, the issue came when I first went to create a job and I noticed VEEAM wanted to use the 'default' backup repository which it decided was going to be on the D:\ drive...

 

But I ignored that and I found out how to create my own (let's say on the X:\ drive) and that all completed OK (I think), but when I looked I found it must have created a few folders on the D:\ drive during the install...

 

If VEEAM thought I was going to put the backup on the D:\ drive, did it put all those folders on there for that reason?

 

 

If I take my backup repository drive (X:\) away and try to test restore it on another computer, will it say 'can't find xyz file in folder VBRCatalog', or 'can't start restore because setup.exe is missing from folder 7230c5c96a3aa8856644b7'...?

Edited January 11, 2023 by Koldov

[Koldov]

Anyway, answered a couple of my own questions...

 

You can safely remove AWS, Azure, Google, Mac, Linux plug-ins etc.

 

https://helpcenter.veeam.com/docs/vbgc/vbr_integration/uninstall_plugin.html?ver=30

 

This should stop the error in the Server's Event Viewer (Veeam.GCP.PlatformService.WebApi.Framework.Common.Middleware.CUnhandledOperationsMiddleware).

 

You can move the location of the VBRCatalog:

 

https://www.veeam.com/kb1453

 

So, just the weird folder (7230c5c96a3aa8856644b7) and the ProgramData ones to find information on...

 

When I click on 'Files' nothing appears in the 'working area' no .VBK files, but maybe because I haven't enabled "guest file system indexing" or something, so I'm a bit lost. However, I did go through a partial restore and it does at least show the other 'no label' partitions, so I guess it is backing them up in the job.

 

EDIT: ProgramData folder holds the IRCache folder which is for Instant Recovery for VSphere VMs:

 

https://helpcenter.veeam.com/docs/backup/vsphere/install_vbr_data.html?ver=110

 

So not needed for HyperV I think.

 

EDIT 2: It looks like the random letters and numbers folder may have been created during install as it seems to hold a lot of setup, x64, redist, resources, SQL folders.

Edited January 11, 2023 by Koldov

[jthompson]

When you add a VM to a Veeam backup job, it will by default include all of the disks attached to the VM, even if they're not mounted in the guest OS at the time. If your VM was a file server that had e.g. 8 VHDX files attached to it, all of those disks would get included in the VM backup, and any disks added at a later time would automatically be included. Veeam just sees a VM with some disks attached, rather than working from withing the guest OS and seeing "OS C, Staff Share E", etc.

 

If you wanted to only include specific disks, go to edit the Veeam job and in the Virtual Machines list for it, select the VM and click on Exclusions > Disks > VM > Edit.

[DavR]

My experience with the ProgramData folder that randomly appears is that this is a temporary file location, that is usually empty, but might be used during the backup process.

 

I think Veeam just chooses whichever volume has the most free space available at install time and dumps that temp directory there. it's untidy, but I've ignored it, and its not given me any problems.

Edited January 11, 2023 by DavR

[Koldov]

Does anybody know which options to choose to just do a full backup every night?

[DavR]

Does anybody know which options to choose to just do a full backup every night?

 

Backup full/incremental options are under the Advanced button, on Storage page when you edit the job.

 

Full backups every night sounds a bit extreme, any particular reason? Veeam is heavily geared towards incremental backups.

 

That said, there are options to create full backups periodically ("active full" in Veeam terms). You could try just choosing weekly, and ticking every day of the week that you want a backup. You'd need to test to see whether this would replace the standard incremental, or run as well as, though.

[Koldov]

Yeah, not only geared towards incremental, but it seems that is the default option (actually it doesn't even give you an option to change)...?

 

On the 'Storage' page of the job there is no full/incremental choice to be made that I can see, just the 'Backup Repository' value and 'Retention Policy' and in the 'Advanced Tab' the option for creating 'Synthetic Full' (?) or 'Active Full' periodically (I doubt this would be instead of, but it would be in addition to the incremental).

 

 

It seems there is some way to do it, but it is complicated/convoluted... as you say it's not really designed to do it and nobody understands why you'd want to do it, so I guess I am in the wrong.

 

https://forums.veeam.com/veeam-backup-replication-f2/setting-up-a-recurring-full-backup-with-the-backup-copy-job-t74416.html

 

I'm just the opposite I guess I don't really see the need for everything to be incremental backups. I know it's a thing and means less time and resources, but I have plenty of both. These aren't file servers that I'd want to restore an individual file from last week, they are purely the VM Host, VM Print Server, VM DC - DR backups that would be used as soon I'd recovered from whatever disaster happened! I just want to know the latest backup is a full backup and when I put the disk in to restore, it will be the only thing I need to worry about.

 

What I don't have is the ability to stop the corruption or error on a single incremental backup meaning the whole chain is compromised, so this sort of thing worries me (just swap 'delete' for 'corrupt'?)...

 

"To be able to restore data from a Veeam Agent backup, you need to have a full backup file and a chain of subsequent incremental backup files on the disk. If you delete a full backup file, the whole chain of incremental backup files will become useless. In a similar manner, if you delete any incremental backup file before the point to which you want to roll back, you won’t be able to restore data (since later incremental backup files depend on earlier incremental backup files)."

 

I haven't got VEEAM on to the Big Boys yet (SIMS Server, File Server, FMSO DC, WSUS) most of those are WSB have been full every night for years (the exception being the file shares because just one of those is +300GB and you're right, a full backup of that every night would be a bit extreme).

[DavR]

What I don't have is the ability to stop the corruption or error on a single incremental backup meaning the whole chain is compromised, so this sort of thing worries me (just swap 'delete' for 'corrupt'?)

Honestly, I don't think that's a big risk. But I understand why you'd be concerned about it.

 

If you just want a single, reliable, file to restore from, you could always set the retention to just one backup. I used to do that with some archive data I had, and didn't need a history of restore points, I think I did a reverse incremental. That way, you've always got one single file of your latest data, that won't suffer from backup chain failure.

[jthompson]

With incremental jobs, it does mean that you can run one on demand to quickly back up a server to that point in time within a couple of minutes or so, ahead of installing some big updates or clicking some other scary button.

 

If you have plenty of resources, perhaps you could double up and have a daily incremental job, with a separate job scheduled to do a full later on. That's assuming Veeam is okay with multiple jobs for a VM.

[Koldov]

I think the options you are thinking of are in the VM backup job (I have enabled 'Reverse Incremental' in the jobs for the VMs), but I don't see them in the job for the physical VM Host...

 

I have also thought about the 'One Backup To Rule Them All' I even thought about scripting the delete of the old backup before it took the new one, but it would be just my luck for it to fail the new one and I'd have nothing!

 

I hadn't thought about using the VEEAM retention thing being set to one day, but I wonder when it deletes the 'old' version on the disk or maybe it just seamlessly blends them...

 

I also thought about trying the 'Schedule' of the (incremental) job for a Sunday and then set 'Storage > Advanced > Backup - Create active full backups periodically' to Mon - Fri... but I'm not sure that will override the 'Schedule' and all I will get all week is one incremental of nothing on a Sunday?

Edited January 12, 2023 by Koldov

[Koldov]

With incremental jobs, it does mean that you can run one on demand to quickly back up a server to that point in time within a couple of minutes or so, ahead of installing some big updates or clicking some other scary button.

 

Would you do that through VEEAM rather than a checkpoint/snapshot on the Hyper-V...?

 

Is it quicker/better/easier/more consistent to, A) do, and B) restore (genuine questions).

Edited January 12, 2023 by Koldov

[jthompson]

It means that you don't have a Hyper-V checkpoint hanging around that would need a merge operation to get rid of. I haven't really used VM checkpoints for ages, so perhaps not best placed to properly answer your question.

 

I guess I find the sea of green in the successful Veeam backup email notifications soothing and reassuring when faced with having to click on a scary button.

[DavR]

I hadn't thought about using the VEEAM retention thing being set to one day, but I wonder when it deletes the 'old' version on the disk or maybe it just seamlessly blends them...

 

Now I think about it, setting the retention policy to one restore point WON'T lead you to a magical, backup chain free, existence. It will still keep the chain of incrementals going back to the last full. Eg, if you run a synthetic full every weekend, that would be one retention point, but it will still do incrementals during the week, and keep the chain for as long as that last restore point needs it to refer backwards.

 

For reverse incremental, it's slightly different. It starts with a full backup file, does an incremental, then merges the two to create a single latest full backup.

 

You say you're blessed with time and resources on this backup, could you run the usual incremental backup, but also tick the box to create Active full backups periodically? In here you could tick weekly, and tick the days of the week you want them. So yes, you're backing up twice, but, if you've got the time and resources, it will give you best of both approaches.

[Koldov]

I haven't really used VM checkpoints for ages, so perhaps not best placed to properly answer your question.

 

So you have found restoring full servers (VMs I presume) from VEEAM backups quicker/easier than Hyper-V. I've never really had to do either in all my time in IT (so it scares the life out of me), but I know it's something I need to get confident with doing...

 

Now I think about it, setting the retention policy to one restore point WON'T lead you to a magical, backup chain free, existence. It will still keep the chain of incrementals going back to the last full.

 

I don't know, I think you're onto something, anyway I also still have WSB running, so I have time to experiment with VEEAM.

 

The other thing with having one full backup everytime (rather than full plus x amount of incremental), is that for offsite storage I can use smaller, faster (and relatively cheaper) SSD disks to rotate rather than the big clunky Enterprise TB disks we have now. Thinking about it, I'm not actually sure how VEEAM deals with rotating disks for one backup job, I didn't see any options for it like in WSB. I don't know if it would realise the change in what it had incremented to the last disk and the difference to what it needs to increment on the next...

[jthompson]

So you have found restoring full servers (VMs I presume) from VEEAM backups quicker/easier than Hyper-V. I've never really had to do either in all my time in IT (so it scares the life out of me), but I know it's something I need to get confident with doing...

 

I've certainly found restoring entire VMs from Veeam nice and easy. I think I've only ever rolled back from Hyper-V checkpoints/snapshots when building some linux servers that I wasn't sure on the procedure for. Even then, that was years ago now.

[Koldov]

I'm presuming I need the Hyper-V role and Veeam B&R installed on another computer to test restore?

 

If so what is then the best option in Veeam to do this and also keep it from becoming live on the network?

 

EDIT: Is there anyway to test restore the Veeam backup of the physical host?

Edited January 13, 2023 by Koldov

[DavR]

I don't think you need Veeam B&R installed on the physical host you want to restore to. Just a vanilla machine with Hyper-V role should do it.

 

There are a number of options you can choose on restore to prevent the test restore VM clashing with the live instance - https://helpcenter.veeam.com/docs/backup/hyperv/performing_full_recovery.html?ver=110. Easiest to my mind would be to restore to the test host, but don't power on VM as part of the restore process. Then, disconnect the network card in Hyper-V settings, and you can boot up your test restored VM off the network.

 

As for physical host, I didn't know Veeam could back them up tbh, only ever used it for VMs. My restore strategy these days is to recover the host manually, as there shouldn't be anything on there bar OS and Hyper-V.

[CHiLL]

The other stuff that Veeam installs, I just left installed. It wasn't taking up that much space and who knows if I may need the utilise them in the future and then can't because I've removed the features.

 

I believe you need a specific license to allow you back up both physical and virtual machines. Our perpetual license only entitles us to virtuals, so we use the free community edition on another server for our physical DC and both our technician workstations. I don't know how this works via VEEAM's new VUL though.

 

Our VEEAM server is installed on a VM (VMware), which has 2x disks; 100GB OS disk and 100GB data disk. Veeam is installed onto the OS disk, with the default repository pointing to the data disk. I have another repository on our QNAP NAS that is used as our main backup repository.

 

Back in the day, I used to like doing reverse incremental backups, as it means I always had the latest full backup file every night, which could be easily copied off for off-site backup. However, with ever-increasing file storage requirements, we were finding that our backups were running through the night and into the working hours of the morning too, impacting user performance accessing files, server speeds, etc. Instead, we now use forward incremental, but with weekly synethetic backups for each job (I've spread the synthetic backups on different days throughout the week to reduce workload/them all running on the same day). The benefit of having a synthetic backup means that I always have a full backup file dated within the past 7 days. We can just copy off these .vbm and .vbk files to our external storage (sure it's not of the day before, but it's only for disaster recovery and I think losing up to a week's data is less of an issue than recovering from a whole-site disaster!)

 

For the jobs themselves, we have all VMs as separate jobs instead of jobs that contain multiple servers, as this creates a seperate backup files per server instead of one large backup file per server. Our rationale for this, is should we need to recover data from our off-site backup, we only have to download the server we need, instead of a massive file containing all servers. It also increases the speed of a disaster recovery, as you can prioritise the servers you download, such as the DCs, instead of downloading all server, including non-vital ones, such as web servers for booking systems, management servers, remote access, etc.

 

For scheduling, we have all jobs set to start at 10pm, though limited the amount of concurrent jobs allowed to run to 4, so they all aren't running at once. Instead, once one of the 4 concurrents finished, Veeam automatically starts the next job and so on. This, combined with incremental backups has massively improved our performance and backup completion times, so they're well finished before the school is opened the next morning.

 

Email notifications I have set to warning/errors only, as I don't want to de-sensitise myself if a job has a warning but it's lost with all the other sucessful email notifications.

 

I know there's a fair amount in this post, not all is what you asked...but I hope it helps further inform the way Veeam can work!

[KK20]

VEEAM can back up physical hosts and could do so with the community version: I used to do this before I licensed for more features and VMs a few years ago. Physical hosts used to consume two licenses though rather than 1 for a VM, that could have changed.

[Koldov]

I don't think you need Veeam B&R installed on the physical host you want to restore to. Just a vanilla machine with Hyper-V role should do it.

 

Interesting, but I can't quite see how that works... I have my backup disk connected to my work PC now and all I see are 3 files (.vbm, .vrb & .vbk - I presume a backup, an incremental and a catalogue of change or something I guess), all proprietary Veeam file types, so nothing I can use without the software installed.

 

Although, I have seen something about copying some files from the Veeam install to allow an extraction from the backup and I guess all the relevant files might be in there (Hyper-V disk, machine, config) and then somehow add them into Hyper-V?

 

There are a number of options you can choose on restore to prevent the test restore VM clashing with the live instance - https://helpcenter.veeam.com/docs/backup/hyperv/performing_full_recovery.html?ver=110. Easiest to my mind would be to restore to the test host, but don't power on VM as part of the restore process. Then, disconnect the network card in Hyper-V settings, and you can boot up your test restored VM off the network.

 

Thanks, I'll see what the options are on the restore through Veeam but I have other VMs running so don't really want to disconnect from the Network card in the Hyper-V global settings.

 

As for physical host, I didn't know Veeam could back them up tbh, only ever used it for VMs. My restore strategy these days is to recover the host manually, as there shouldn't be anything on there bar OS and Hyper-V.

 

Yes, have thought similar recently as trying to move everything onto VMs and technically I suppose that's the way to go, but I have some 'messing about' to do (changing the host from MBR disk to GPT, partition deletes etc) so I want to make sure the backup works anyway. I'm just not brave enough to export the VMs, wipe it, install OS & updates, mess about with I.P.s and host names, add Hyper-V role, import machines back and expect everything to work... although I guess in theory it should be as simple as that... I'd want downtime to be minimal as one of the VMs is the PDC (though I guess I could transfer roles to another DC and demote it)...

 

EDIT: HAHAHA! Never mind... I installed Veeam B&R... and after all that I can't do a 'Full Restore' into Hyper-V on a 'Windows 10 machine', it says I can do an 'Instant Restore' though and then fails that due to Windows 10 not having a Hyper-V VSS Writer...

 

Trying the 'extract' route now... There's obviously no 'options' here so I'll try it creating a virtual machine with the network disabled and import the disk (because I don't think it contains the machine settings in a form Hyper-V can use to import a machine). I don't know..."Do backups"... They say.... OK... "Test backups"... They say... HOW?! Nobody seems to actually know... I guess if I get more confident with Veeam I will try restoring it on the actual server (New name and UUID with no network is the way I guess?) and hope I don't make any stupid mistakes (unlikely as I'm a bit new to VMs and Veeam)...

 

The host backup couldn't restore to a VM (Instant Restore to a VM - P2V)

 

https://helpcenter.veeam.com/docs/backup/agents/agents_restore_hv.html?ver=110

 

to test either though which was a function I read Veeam can supposedly do (Failed to get external virtual switches. [WMI] Empty result)... So I think I can try to export that as a virtual disk too, but I don't have the space on Hyper-V SSD I use in machine for the server disk... I can view the contents though and it seems to have files in there so that's a bonus...