Windows Server Backup / VHDX performance

[dayzd]

Hi all

 

I'm having some difficulties with Windows Server Backup and/or VHDX performance on my new(-ish) WS2012R2 systems. I have 6 virtual servers on two hosts, and I'm really struggling with backing them up in a timely fashion.

 

For budget-based reasons, I'm using Windows Server Backup. Each server runs a daily differential to a dedicated local VHDX backup disk, and a manually scheduled weekly full backup to a UNC share. For the most part, this has been working just fine.

 

Lately though, the backups on my file server have been incredibly slow. For example, last night's backup defaulted to FULL instead of incremental (as apparently my weekly fulls haven't run correctly for a couple of weeks!), and after 9.5 hours it had backed up just 159GB which is by my reckoning a paltry 4.6MB/s. This is from one fixed-size VHDX to another, stored on the same 12-disk RAID6 array, but different LUNs.

 

 

Have I made some schoolboy errors while setting up my VHDXs? Could a poorly chosen block-size have impacted on this? I'm beginning to think I've made some mistakes, as disk speed isn't stellar for the whole system. I know some of this is due to the RAID6 array, but as I'm a one-man-band here, I wanted redundancy. Was this a mistake?

 

 

I know it's difficult trying to troubleshoot vague speed issues, but if anyone has any suggestions, I'd appreciate a hand!

[Sagima]

If you can do a manual file transfer between the two LUNs and you get the same speed then presumably it is the disks. If it is much faster (and even on raid6 it should be) then it's something that WSB is doing. Is it only backing up the VHDX files that takes so long or is it just that they are so big you notice that it is taking a long time.

[dayzd]

Hi @Sagima

 

Yes, doing manual file transfers between LUNs is a lot faster - something in the order of 400MB/s.

 

I'm not directly backing up the VHDX files, though. User files are inside one VHDX on one LUN, the backup target is another VHDX on another LUN. To the server, they're both local disks. Within the VM, a manual transfer of files between the Data VHDX and the Backup VHDX peaks at around 40MB/s.

 

 

Yes, part of the reason that I'm noticing is the size of the disks - the data disk is 1.5TB and the backup target disk is 2.5TB. The actual data being backed up is around 700GB. All my other servers have the same speed issue, but as they're backing up far less, they still complete in a manageable amount of time.

 

 

I'm aware WSB is not known for it's speed, but looking at the numbers above (as reported by Windows' file copy dialog box), there is surely something amiss, either with my VHDX's or WSB.

[Sagima]

From that I think my first thing would be to check the virtual network adapter and make sure the hyper-v integration tools have been installed in the virtual machine and you are not using the legacy adapter for your network connection. 2ndly check that the snapshots/checkpoint directory isn't on a slow disk and that ideally you don't have any snapshots/checkpoints.

 

If I think of anything else I'll post again

[dayzd]

Sagima, Thanks for your responses.

 

Unfortunately, I can't check up on anything you've suggested there - the LUNs are connected to the Hyper-V Hosts via fibrechannel, so I've no network adaptors to adjust. Integration tools are installed by default in 2012R2 and I'm pretty sure they're up to date and I've not been using any snapshots or checkpoints.

 

I'll keep on Googling for a bit longer!

[Sagima]

I meant the virtual network adapters and integration tools of the clients (that which connect the clients to the host server) not the connection between the host server and lun. I may be mis-imagining your setup but in my mind it is

Client

(Slow Connection - virtual network)

Host

(Normal speed connection - fibre channel)

LUN

 

The only other thing that comes to mind is the disks themselves being differencing disks rather than expanding or fixed sized. I'm told differencing disks are quite slow compared to fixed but I've never used them.

[dayzd]

The client VMs don't access the ClusterStorage via network adaptor, do they? Surely Hyper-V passes them the VHDXs directly?

 

Ether way, all but one of the VMs are Gen2 / 2012R2 and are not using Legacy NICs. They're all the default 10Gbps virtual adaptors with VMQ turned off.

 

All the VHDX files I'm using for all my servers are fixed, except for the Data disk on the file server that I'm struggling with. The reason that is dynamic is because when deploying these servers, I found the initial fixed-size disk I created to perform very slowly. When re-created as dynamic, it was suddenly faster.

[Sagima]

I was thinking that WSB backing up the contents of a client (or transferring contents from one client to another) would access the client via its network adapter - maybe try changing it to the legacy adapter and see if it makes any difference to the file transfer speed between the two. The 2nd hotfix on this page may be relevant to you Recommended hotfixes, updates, and known solutions for Windows Server 2012 and Windows Server 2012 R2 Hyper-V Network Virtualization (HNV) environments depending how your server is set up.

It's been so long since I've used WSB that it may be different in how it backs up hyper-v machines to how I think it would.

Edited March 31, 2015 by Sagima
forgot a bit (in brackets)

[dayzd]

WSB is running inside the VM, copying from one local disk to another (the two VHDXs I mentioned above) - it won't be using a network adaptor.

 

From what I can tell, my speed issue is purely disk related, not network.

 

 

 

 

(Edit: I'm aware I mentioned a file-share backup in my original post, but I know that's going to be slow and isn't what I'm trying to resolve here.)

Edited March 31, 2015 by dayzd

[Sagima]

Right I'm with you now. Sorry I thought you were running WSB on the host and backing up from one VM to the other, with their respective storage being on different luns

[dayzd]

Strap in, here comes a big update!

 

Since I couldn't figure out what was going on, I got our support company in. They didn't deploy the kit I'm working on, but they did spec it, so they can lend a hand troubleshooting! They sent an engineer out to examine the setup and help me find any bottlenecks or performance problems.

 

On-site

 

Happily for me, the engineer didn't at any point look at my Hyper-V cluster and go "Why the hell did you do that?!". There were a couple of things he said he may have done differently, but all for management reasons, not performance.

 

The engineer did note that when logging in to VMs, System Center would take quite a while to load up, where on most systems he was familiar with it appeared very quickly, which added to my feeling that overall performance isn't as expected. Given that day-to-day, everything runs fine and the backups are the only really troubling part of the system, we focused on that.

 

After looking through as much as we could in the setup of my Hyper-V hosts, shared storage and client VMs, plus finding out as much as we could regarding Windows Server Backup logs (there aren't any beyond 'job started', 'job finished'), we really hadn't got anywhere. We agreed some tests for me to carry out to try to understand what slows down WSB.

 

Testing

 

The time it takes for System Center opens upon first login around doubles when the VM is placed on the SAN vs the local storage on the Host Server. This time also increases once the server is added to the domain, but this is to be expected to some degree.

 

Testing WSB, I used a copy of the base server image all my production servers were imaged from, and a fresh VM with a new installation of 2012R2. For the first two batches of tests, I put the same amounts of data in each VM for it to back up – in the final batch, I used more data on the new VM.

 

None of the tests showed a great variation in speed between the two VMs, or between the total volume of data backed up.

 

The fastest backup speeds were had when backing up either VM when stored on the Host’s local disks, to a VHDX stored on the SAN, accessed via UNC path to the same host (120-150MB/s).

All other tests (with the VM stored on either Host local disk or SAN, backing up to USB3 external drive or VHDX stored on the SAN, accessed via UNC path) returned roughly the same speed (40-60MB/s).

 

Conclusion

 

All of these tests, including those that directly mirror the setup of our production servers (SAN to SAN), backup at a much faster rate than the production servers I'm struggling with.

 

The only thing I've been unable to test is an extremely large backup (hundreds of gigabytes) as I simply don’t have the capacity!

 

Moving on...

 

I'm not entirely sure where to go with this now. I've proved that the system I have can backup data at a decent enough rate that it won't cause any trouble - my file server currently manages 676GB in 39 hours. Even at the slowest speed recorded by my tests this would be reduced to less than 5 hours.

 

Short of moving to a new backup solution, I'm at a total loss. Right now I don’t have the funds to buy anything new, so if anyone can suggest anything that might help with my disk speed troubles it really would be appreciated.

[dfergusson]

Am I right in thinking that your backing up to same SAN on the same vdisk?

Do you do any further backing up from that to another device?

 

Regarding your issue have you tried disabling ODX on the hosts? It enabled by default on Windows 2012 r2 see if that makes any difference.

[dayzd]

@dfergusson,

 

Yep, all my SAN disks are RAIDed into one big vdisk. It was set this way to get the most space from the disks I have, but I've since realised it might have been a little more flexible set up another way!

 

No. Once the VMs back up to their locally attached VHDX (stored in my SAN), that's it. A full remote copy is made of each server once a week via UNC share - a different server runs each night. All my backups are staggered - unless something overruns, only one server is backing up at a time.

 

Unfortunately for me, my cheap little HP P2000 G3 SAN doesn't support ODX. Is the fact that it is enabled by default likely to cause an issue?

[dfergusson]

 

Yep, all my SAN disks are RAIDed into one big vdisk. It was set this way to get the most space from the disks I have, but I've since realised it might have been a little more flexible set up another way!

 

 

One big vdisk seem to be norm these days, it is how I've setup mine up.

 

No. Once the VMs back up to their locally attached VHDX (stored in my SAN), that's it. A full remote copy is made of each server once a week via UNC share - a different server runs each night. All my backups are staggered - unless something overruns, only one server is backing up at a time.

 

I'm don't fully understand the above.

 

Unfortunately for me, my cheap little HP P2000 G3 SAN doesn't support ODX. Is the fact that it is enabled by default likely to cause an issue?

 

ODX enabled on host can course issues and as you said is not Supported under P2000 might be worth disabling it.

[dfergusson]

From what I understand your VMs have two VHDX attached to them, each placed in different Luns on the same Vdisk.

And using Windows backup within each VM to back from one VHDX to other.

 

What you have effetely done is having single Hard drive in a pc with two partitions copy all your file from C to D partition.

Doing this on Raid6 is going to impact this further.

 

If you want faster backups look at backing up from host utilising the VHDX Directly (Single Large File) vs 100k's+ little files within VM to same location.

Second set the back destination to disks that not being used as the read source.

Edited April 17, 2015 by dfergusson

[dayzd]

No. Once the VMs back up to their locally attached VHDX (stored in my SAN), that's it. A full remote copy is made of each server once a week via UNC share - a different server runs each night. All my backups are staggered - unless something overruns, only one server is backing up at a time.

I'm don't fully understand the above.

I've used the schedule of WSB to do daily incremental backups to each VMs second disk.

I've also manually added a scheduled task to run wbadmin once a week to make a complete backup to a geographically remote share on the same network.

 

From what I understand your VMs have two VHDX attached to them, each placed in different Luns on the same Vdisk. And using Windows backup within each VM to back from one VHDX to other.

Yep.

 

What you have effetely done is having single Hard drive in a pc with two partitions copy all your file from C to D partition. Doing this on Raid6 is going to impact this further.

That's one way to describe it, yes. If I were doing this with a single disk, I'd have no-one to blame but myself for the 5MB/s transfer speed I'm seeing, but in a 12-disk raid, of any kind, that figure is just ridiculous.

 

If you want faster backups look at backing up from host utilising the VHDX Directly (Single Large File) vs 100k's+ little files within VM to same location.

My Hyper-V setup uses CSVs. WSB on the host does not support CSVs. Also, since Windows Server Backup works at block-level not file level, this does not result in faster backup times vs individual files - something I've proven in testing.

 

Second set the back destination to disks that not being used as the read source.

Given how far I've got (I have 7 production VMs on this hardware), re-arranging disks is not possible. Besides this, my tests have already proven that it can work faster in this configuration than I'm seeing with any of my production servers: when stored on the SAN in one LUN, and backing up to the same VDISK in another LUN, I got speeds of 55, 56 and 58 MB/s. These aren't stellar numbers I know, but the tests were carried out during the day while all my users are online, and they still represent a 10x increase in what I see from my production servers.

[PotNoodleTech]

Sounds like a pure disk speed issue rather than an issue with the actual backup software/configuration.

 

What SAN hardware, how is it connected to the hosts, what disks are you using, what raid configuration?

[PotNoodleTech]

Also a handy way of quickly testing disk performance is to download the free zipped version of CrystalDiskMark, unzip it then pop it on the drive of the VM you wish to test - it runs a quick synthetic benchmark post that back up here and that will give us an idea if there is an inherent storage subsystem bottleneck that the backup regime is highlighting.

[dayzd]

Hi @AButters

 

I've not used CrystalDiskMark before. Kinda handy, that!

 

My SAN is set up with a 10-disk (7200rpm) RAID6 vdisk, plus two hot-spares (as a one-man shop, I'm paranoid). Everything lives on that, but the vdisk is chopped up into separate LUNs for data management. Connection to the VHOSTs is SAS.

 

These are the results I got from all of my VMs. The two VHOST entries at the bottom are for reference (they have a simple RAID1):

 

The write tests are fairly consistent, but the read tests are a bit all over the shop! Possibly due to these servers all being in production, but I'm not sure. What do you think?

[dfergusson]

Hi @AButters

My SAN is set up with a 10-disk (7200rpm) RAID6 vdisk, plus two hot-spares (as a one-man shop, I'm paranoid). Everything lives on that, but the vdisk is chopped up into separate LUNs for data

 

What module name/number of those drives that make up your RAID6? Physical size 2.5 or 3.5? SAS or SATA?

[dayzd]

What module name/number of those drives that make up your RAID6? Physical size 2.5 or 3.5? SAS or SATA?

HP 1TB 7.2K SATA 3.5 (731690-001)

[Boredguy]

Sata Agh

[dayzd]

Sata Agh

Helpful.

 

It's what I was spec'd by my support company and what fitted in budget. And they weren't cheap either - if they're poop drives, I'm gonna get annoyed!

[dfergusson]

You don't use SATA for Critical/Production servers, they should be reserved for Desktop or backup servers.

 

Your backup speed is what I would expect based on your hardware and setup.

With a 10 disk Raid6 using SATA and 7.2RPM drive you looking at about 250 IOPS at 50/50 Read Write.

Edited April 21, 2015 by dfergusson

[Boredguy]

For server storage other than archival data, I'd have gone with SAS drives as they generally handle faster read/write speeds than SATA, but if it was budget limited I can see why you went for them.