CHiLL Posted October 15, 2019 Posted October 15, 2019 We use a QNAP TVS-1271U-RP as our on-site backup storage. It currently has 11x 3TB Seagate ST3000VN0001 NAS HDDs (7,200RPM) drives in a RAID5 configuration with a CIFS share and 4x GbE ports trunked to a 4GbE connection. The access speeds have always been dire when VEEAM backs up to it and it is currently running between 3-10MB/s and reports the bottleneck as the target (QNAP). In comparison, I did 5 file write speed tests on the QNAP and our HP MSA used for our production data (7,200RPM MDL drives). The average results were: QNAP Write: 12.646MB/s QNAP Read: 65.142MB/s MSA Write: 53.792MB/s MSA Read: 156.996MB/s The disk health reports Good in the QNAP console. I cannot figure out why it's so bad.
PotNoodleTech Posted October 15, 2019 Posted October 15, 2019 Check if the QNAP has battery backup write back cache and that it is enabled in your array config. I suspect it doesnt. What settings are you using for the backup copy job? I personally only buy Synology and have never had a bad experience other than the odd seagate drive developing bad sectors mine have been going strong for years. I set it up simply as 12 drives in RAID6 and only a single 1GB lan port, no trunking or anything.
CHiLL Posted October 15, 2019 Author Posted October 15, 2019 (edited) Check if the QNAP has battery backup write back cache and that it is enabled in your array config. I suspect it doesnt. What settings are you using for the backup copy job? I personally only buy Synology and have never had a bad experience other than the odd seagate drive developing bad sectors mine have been going strong for years. I set it up simply as 12 drives in RAID6 and only a single 1GB lan port, no trunking or anything. [ATTACH=CONFIG]55342[/ATTACH] Hm, I can't find anything along those lines in the QNAP settings. Our backups are configured as reverse incremental (which I understand are much slower than standard incremental, but not as slow as we're seeing!). The reason for this is because it means that the latest backup file is the full backup. So we only need to copy that and the metadata file to offline or cloud storage, rather than copying the full backup, metadata and all subsequent incremental files. Edited October 15, 2019 by CHiLL
PotNoodleTech Posted October 15, 2019 Posted October 15, 2019 Ok I'll double check my settings tomorrow. I think I switched back to normal incremental with a weekly full backup. I know that reverse incremental significantly slows down the backup speed on a backup copy job as it has to read the data from the NAS to the Veeam Server at the same time that it writes the new data back to the NAS server. So you're effectively doubling your reads and writes?
southhamster Posted October 15, 2019 Posted October 15, 2019 QNAP NAS can support an SSD as a cache. Is it worth trying one if you have spare slot in the NAS?
CHiLL Posted October 15, 2019 Author Posted October 15, 2019 QNAP NAS can support an SSD as a cache. Is it worth trying one if you have spare slot in the NAS?I did see that, though only slots 1,2,3,4 are capable of using an SSD as a cache drive according to QNAPs documentation. All those slots are currently in use. I do have a couple of spare slots at drive bays 11,12, but I'd have to do some drive bay swapping. How does this work? I know the drives are hot-swappable. Can I remove a drive from bay 1 and move it to bay 12? Or would that trigger a RAID rebuild? Also, what capacity of SSD would be suitable? I have a spare 120GB Kingston A400 drive in the office. Would that be sufficient? Or would I need a higher capacity or more of them. Our QNAP has 8GB of RAM.
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 Looks like our QNAP TVS-1271U-RP has two mSATA slots on the motherboard that can be used for SSD caching. That would solve the issue of having to shift drives around. Would I need to utilise both slots? What capacity drives would I need? Our largest backup file is approximately 2TB and we are using approximately 18TB of storage in total on the QNAP.
PotNoodleTech Posted October 16, 2019 Posted October 16, 2019 SSD caching is not really going to help so much in a backup scenario as when you are writing the backup to the NAS you are doing it in one big stream, which once it fills the SSD up, it will then slow down to half the speed of the native SATA speed for the duration of the rest of the backup. So you may get fast backup for like 30 minutes and then it'll go back to even more super slow than before whilst it continues writing from the SSD cache to the HDD array whilst still trying to write the rest of the backup job to the HDD array at the same time. I would concentrate on addressing why you are getting 12MB/S write speed out of a £4000 NAS?
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 (edited) SSD caching is not really going to help so much in a backup scenario as when you are writing the backup to the NAS you are doing it in one big stream, which once it fills the SSD up, it will then slow down to half the speed of the native SATA speed for the duration of the rest of the backup. So you may get fast backup for like 30 minutes and then it'll go back to even more super slow than before whilst it continues writing from the SSD cache to the HDD array whilst still trying to write the rest of the backup job to the HDD array at the same time. I would concentrate on addressing why you are getting 12MB/S write speed out of a £4000 NAS? Ah, nevermind then! I have absolutely no idea why it's going so slow and I don't where to start and as far as I can tell, it's configured correctly. Checking the transport mode within VEEAM, it's configured as Virtual Appliance, which I presume is correct as they're VMs on ESXi hosts. I'm considering getting a third party support company to take a look (we have a points contract we need to use up anyway). Edited October 16, 2019 by CHiLL
mavhc Posted October 16, 2019 Posted October 16, 2019 Can you do a speed test on the device itself? Not via the network.
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 (edited) Can you do a speed test on the device itself? Not via the network. I can only do sequential read and read IOPS tests and they look good - so the issue must be with either the QNAP configuration or network? I doesn't seem to be VEEAM as a file copy [ATTACH=CONFIG]55352[/ATTACH] Secondly, a Windows file copy from the VEEAM VM server to the mapped QNAP drive (where VEEAM backs up to) has transfers at 50-60MB/s: [ATTACH=CONFIG]55353[/ATTACH] Maybe the transfer speed applicaiton I was using is crap. I'm going to use DiskSpd, which is recommended by VEEAM for testing. Edited October 16, 2019 by CHiLL
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 Jumbo frames on the QNAP? Jumbo Frame isn't a yes/no option, it's just a value box. It's currently set at 1500.
pantscat Posted October 16, 2019 Posted October 16, 2019 Indeed - 1500 isn't a jumbo frame. I suspect that might be causing some performance issues. 9000 is a more common value, but make sure your switching infrastructure can support it.
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 (edited) Indeed - 1500 isn't a jumbo frame. I suspect that might be causing some performance issues. 9000 is a more common value, but make sure your switching infrastructure can support it. The VM host (10GbE) and QNAP (4GbE trunked) are both connected into our HP ProCurve 5406zl core switch, which does apparently support jumbo frames. However to enable it, it's enabled on the VLAN the devices are on. They're both on the same VLAN, but it's also the same VLAN as all our other server infrastructure. Is enabling jumbo frames on that VLAN going to use issues with other services/servers that aren't configured for jumbo frames? Edit: After connecting to the switch via PuTTY, it looks like Jumbo frames is already enabled on that VLAN. Edited October 16, 2019 by CHiLL
pantscat Posted October 16, 2019 Posted October 16, 2019 In that case, you should have no issue changing the mtu value on the QNAP to 9000. You can always change it back!
CHiLL Posted October 16, 2019 Author Posted October 16, 2019 In that case, you should have no issue changing the mtu value on the QNAP to 9000. You can always change it back! Very true. I will change it today if I can and see what the backup results are tomorrow. Though VEEAM is now health-checking our 2TB staff server backup, which may take some time...so maybe I'll have to make the change tomorrow instead!
CHiLL Posted October 17, 2019 Author Posted October 17, 2019 I changed the MTU yesterday to 9000, however that hasn't made a difference to the performance. I converted one of my jobs from reverse incremental to forward incremental and the speed for that one job has gone from 5MB/s on Tuesday night's backup to 49MB/s at last night's backup (speeds reported in the job summary section in the VEEAM console). The bottleneck also changed from Target to Proxy. Good signs. We use a Robocopy/RClone script on a schedule to copy the .vbk and .vbm reverse incremental backup files to our cloud storage. Forward incremental creates a .vbk and subesquent .vib files and the synthetic full also creates a .vbk file from what I understand. How do I get the script to pick up only the synthetic full? At the moment we just use a wildcard for .vbk and .vbm files. We don't want the script to unnecessarily copy both .vbk files, only the synthetic full.
mavhc Posted October 17, 2019 Posted October 17, 2019 Reverse incremental does sound like a lot of IO is needed, got to read data from the target, change it, and write new data, instead of just writing a new file. Needs more ZFS
CHiLL Posted October 21, 2019 Author Posted October 21, 2019 (edited) So far so good, performance has generally been very good using forward incremental instead of reverse. I have some questions about the synthetic full backups: 1) If I copy the synthetic .vbk and the metadata .vbm file to off-site/cloud storage...are they enough to use as disaster recovery? 2) How do synthetic full backups vary from active full backups? Synthetic full backups cannot be scheduled on a months basis, which I'd prefer. Instead they can only be run on a daily or weekly basis. Edit: So from my research, for a synthetic full, the .vbk file is built on the SAN by using existing backup files in the chain (so mounting the VMs and no network transfer). An active full backup would mount the VM and perform a full backup over the network. So my first question still stands - can I copy off this synthetic full backup file and recover the server fully from it? Edited October 21, 2019 by CHiLL
ass17 Posted October 23, 2019 Posted October 23, 2019 Obviously you can’t change now but RAID 5? really? I’d like to think you were planning to change it one day [emoji1]
CHiLL Posted October 24, 2019 Author Posted October 24, 2019 Obviously you can’t change now but RAID 5? really? I’d like to think you were planning to change it one day [emoji1] Yeah, I'd prefer RAID6 or RAID10, but I don't have any other storage with enough capacity to store the data to destroy and rebuild the array!
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now