Major Issue with VMware's CBT

[Sandman]

Hi all,

 

Just posting to make you aware of a major issue with the way VMware handles the indexing of the ChangeID.

 

Unfortunately the ramifications are rather large...if your chosen backup software does not do a '*' ChangeID backup (i.e. a query for all 'dirty' or used blocks in a VMDK after a 'revert', or a 'goto' in a snapshot tree) and a duplicate "unique" ChangeID is used for the backup then your backups could well be corrupt.

 

There is more information on Tom Howarth's blog site which basically covers the jist of the problem.

 

Major Issue with Change Block Tracking | PlanetVM

 

As far as I know esXpress 4.0 is the only backup product to date that has confirmed a current implemented workaround for this issue.

If a backup taken of a machine has a snapshot esXpress forces the changeID to * what this means it will copy ‘all dirty blocks’. This is obviously a bit slower, but until VMware fix the CBT issue it is 100% reliable.

[rbrambley]

Veeam Software has confirmed that Backup and Replication v4.1 successfully handles this issue without corruption in all but one specific scenario of events. The hotfix has been created for the one remaining scenario and is in "testing for validation" as of this writing.

 

In the above linked thread Veeam also recommends "disabling the use of changed block tracking in the Advanced job settings for all jobs which process VMs where manual snapshot reversal may happen, and triggering Full Backup on these jobs to heal the backup file (in case you believe you may have this scenario happened before for some VMs)."

[rbrambley]

Since I could not add a link in my first post in these forums, I am adding this second post and including the link to the Veeam forums describing the single issue where corruption with CBT enabled incremental backups could still occur. Community Forums • View topic - VMWare issue with CBT ?

[Sandman]

VMware have acknowledged this issue outside of John Troyer's responses by putting up a public knowledgebase here:

 

VMware KB: Reverting to a pre-existing snapshot under specific conditions can cause incremental backups based on CBT (Changed Block Tracking) to become inconsistent