Hyper-V Storage Weirdness..?

[AB_IT]

Hey Guys,

 

Can someone help explain this to me? We have a SAN here shared between two Hyper-V Servers. The Virtual Servers Hard Disks are stored on the SAN in question.

 

We've recently ran out of storage space on the SAN but I can't understand how this is possible as the Hard Drives aren't as big as the Hard Drives on the actual VM... If that makes sense?

 

There also seems to be loads of .avhdx files while are large in size but again I don't understand the reason for this. How can the our 'Data' D: drive be only 211GB large with a maximum capacity of 499GB but then the .VHDX it's associated to be 524GB plus a bunch of these .avhdx files also? Our Userprofile H: drive is another as this is generating huge .avhdx files.

 

Is there anything I can do with this to free up some space or somehow get rid of the avhdx files as not all our servers have these?

 

Sorry if I'm being clueless but I don't want to mess anything up.

 

[Steve21]

So the virtual disk can be larger than the partition in Windows if you've had it previously where you had more data, then shrunk a partition etc, but you might not have shrunk the virtual disk in HyperV (Edit - Also could be you used fixed disks rather than expanding, so it takes the entire space up at the start rather than expanding as needed)

 

The AVHDX are differencing disks that normally occur during a checkpoint, backups etc make these, but they should disappear after the backup completes successfully and merges back into the server/vhdx

 

The easiest way to merge them is normally just by turning off the server (VM wise) and it'll auto merge them back together (although this does rely on you having some space to do that in the first place). Are you fully maxed out?

 

Steve

Edited May 4, 2021 by Steve21

[Guest Guest]

AVHDX files are created when a checkpoint is created (this usually occurs when a backup is taken although can be done manually)

 

Dynamically expanding VHDXs will expand as large as the maximum allowed capacity but don't automatically shrink once the disk space is freed up.

 

https://www.nakivo.com/blog/shrink-compact-virtual-hard-disks-hyper-v/

[AB_IT]

So the virtual disk can be larger than the partition in Windows if you've had it previously where you had more data, then shrunk a partition etc, but you might not have shrunk the virtual disk in HyperV (Edit - Also could be you used fixed disks rather than expanding, so it takes the entire space up at the start rather than expanding as needed)

The AVHDX are differencing disks that normally occur during a checkpoint, backups etc make these, but they should disappear after the backup completes successfully and merges back into the server/vhdx

The easiest way to merge them is normally just by turning off the server (VM wise) and it'll auto merge them back together (although this does rely on you having some space to do that in the first place). Are you fully maxed out?

Steve

Thanks Steve for your help. I've turned the server off and it's taking a while but it says 'Merge in Progress' and has free'd up quite a bit of space again. I'm not maxed out at all, still have a about 200GB free on the SAN storage but when the server makes all these avhdx files on top of it eventually goes over the 200GB that's free and the server pauses.

Our exchange server shares the same storage but this doesn't have any avhdx files at all. Soon we'll be moving away from Exchange to Office 365 and as our exchange server is almost 1TB in size it will free up a lot more room on the SAN as I'll just remove the exchange server altogether once we've moved.

I know it's probably not reccomended but we do have other backup methods. Is there a way I can stop the server in question making avhdx files temporarily and turn it back on at the end of June to stop this from happening or is there no way to do that?

The IT Manager before me setup the server to use a fixed HDD on all our servers so we're kind of stuck and now I don't have the room to convert the HDD type to dynamic and free up a bunch of space. I think he did it because Fixed are meant to be faster? But I had dynamic at my last place and we never had any issues with speed.

EDIT: I tried following this tutorial https://ivankahl.com/disabling-checkpoints-in-hyper-v/ but I don't have the option he has in either HyperV Manager or Failover Cluster Manager I just get the following in both:

 

 

Edited May 4, 2021 by AB_IT

[Steve21]

There must be something that's affecting the backups to make it not merge them though, as it should be as soon as the backup finishes. Unless it's something silly like you have it backing up too often and it's not giving it time to finish merging. You seem to have one at 8:20 and 9:20 as if you're running multiple backups?

 

While you physically "can" disable them, it's really not advisable as they're used to ensure all data is saved if somethings being written during the backup to the disks etc, but as mentioned if it's working properly it should just merge them instantly after the backup finishes.

 

Regarding the fixed disk, you could still shrink them in HyperV (when the servers turned off) but obviously if you shrink them you need to leave space for real expansion. The speed aspect only really is when it's expanding, and isn't a big hit unless you're talking huge SQL queries etc, but it's no harm really anyway, as if it's fixed and you have space it's no different to expanding in that sense

 

Steve

[Steve21]

Just a thought but might be worth having a dig through your event logs on the HyperV host under Hyper-V-VMMS (in events -> MS -> Hyper-V-VMMS etc) and see what it says around that timestamp 1st at 21:35, it should generally log any failures in there

 

Steve

[AB_IT]

There must be something that's affecting the backups to make it not merge them though, as it should be as soon as the backup finishes. Unless it's something silly like you have it backing up too often and it's not giving it time to finish merging. You seem to have one at 8:20 and 9:20 as if you're running multiple backups?

While you physically "can" disable them, it's really not advisable as they're used to ensure all data is saved if somethings being written during the backup to the disks etc, but as mentioned if it's working properly it should just merge them instantly after the backup finishes.

Regarding the fixed disk, you could still shrink them in HyperV (when the servers turned off) but obviously if you shrink them you need to leave space for real expansion. The speed aspect only really is when it's expanding, and isn't a big hit unless you're talking huge SQL queries etc, but it's no harm really anyway, as if it's fixed and you have space it's no different to expanding in that sense

Steve

I have shrank them before but then got a huge telling off by the support company that works for us (soon to be decomissioned) as they where blaming me for the reason the Replica was breaking which I didn't really get as I always ensured I'd start a new replica when I shrunk the HDD. It's not ideal as we replicate our VM's over the internet every 15 Minutes and they reguarly fail, is this to be expected? We have a 200mb leased line at both premises.

 

Okay, noted. Thank you I won't mess with them then, wouldn't want to loose that data or risk it as it's our main file server.

 

I've looked through the logs and can see that the Replication has failed, but there's nothing concerning snapshots or merges or anything like that. Could this be symptom of the replication failing? Other servers replication is okay, just seems to be our biggest two that are having the issues which is our Exchange and File Server.

 

Thank you for helping really appreciate it

 

[KK20]

Just another note, be careful making manual checkpoints on exchange servers. Make sure that your checkpoints for your exchange (or SQL server) VM are set to "production only". Standard checkpoints will most likely barf exchange.

 

You can set VMs to create automatic recovery points, this will auto create checkpoints in intervals. Im sure that hyper-v replication make their own snapshots so if this is failing then these snapshots might not be clearing up after themselves.

 

VEEAM also creates checkpoints, as do most backup software packages. If these are interrupted (or run out of space) then they may not clean up after themselves.

[Steve21]

I've looked through the logs and can see that the Replication has failed, but there's nothing concerning snapshots or merges or anything like that. Could this be symptom of the replication failing? Other servers replication is okay, just seems to be our biggest two that are having the issues which is our Exchange and File Server.

 

Those logs you showed aren't at that date when it first failed though. If you look on your original screenshot the original fail seems to have been the 1st May between 8-9:30~ where there's 2 checkpoints showing

 

Any new checkpoints after that won't clear the older ones that are already broke

 

Would be worth checking back to that 1st one and see what it shows there

 

Steve