BackupExec 12.5- full backups 50 hours!

[lafleur1977]

Hello,

 

I have posted about this before but am still no closer to a solution.

 

I have inherited a backup solution which uses BackupExec 12.5 and I am having trouble with extremely slow backup times. Currently full backups are taking 50 hours including verify and differentlial backups are taking about 25 hours. The backup time has only increased despite trying to decrease the amount of data by implementing quotas (still in progress however) and I could do with a little help to try and get the backup times down to a more realistic time.

 

Firstly I shall explain the solution:

 

Software: BackupExec 12.5 media server running on Windows Server 2008 64-bit. This server is also the main file server.

Hardware: BackupExec 12.5 Media/File server -

Spec: Intel Xeon E5310 1.60 GHz, 8GB RAM and 1 Gbps NIC.

Storage: HP SAN configured by a hardware RAID and formatted as one 5 ish TB NTFS partition. This is used for the main data storage.

HP Disk Array configured by a Windows software RAID (as it wouldn't work in a hardware RAID) and formatted as one 5 ish TB NTFS partition. This is used for backup storage.

HP Tape LT03 drive.

 

BackupExec 12.5 software configuration: Media server has been configured with one daily backup selection list. This currently includes the data files stored on the locally connected HP SAN (currently 1.5 TB), system states for two DC's, system state of an exchange server, exchange database (currently about 80 GB), system state of print server, shadow copy components of print server (not sure what this does and if its needed?), and a few shared folders on a couple of other servers.

 

Two folders have been created on the HP disc array. One called Full and the other Differentials. Both of these have corresponding backup to disc folders under devices. Both were are configured with over right protection of 27 days but recently I have had to decrease these to 3 weeks and 1 week respectively due to low disc space, but I know this can be rectified by increasing the storage.

 

Backup jobs have been created and use the selection list I described above:

 

Mon, Tues, Weds, Thurs Differential daily backups to the Differential folder.

Fri 1 Full monthly backup to the Full folder. These is then duplicated to tape.

Fri 2, fri 3 and fri 4 Differential weekly backups to the Differential folder. These are then duplicated to tape.

 

Conclusion:

 

So the server is configured to do daily differential backups and a monthly full backup. I believe this is suppose to be the fastest method as diffs only backup the changes made since the last full. Full backups are currently around 1.7 TB's and is taking nearly 50 hours to complete, including verify. Diff backups are currently between 100 - 200 GB and are taking about 25 hours to complete, including verify. This also means our daily backups are now overlapping. Duplication to tape is around a few hours and isn't a problem. One thing that I am confused by is the job rate which after checking the job logs varies for each resource. Overall for full backups its around 700MB/min and for diffs is between 70 - 150 MB/min. Backups generally do complete successfully but the backup to disc completion time has done nothing but increase since I started here. I have tried one suggestion from a previous post to increase the .bkf file size as these were set to 1GB but I have increased them up to 50 GB and this only made a minor improvement. I have also tried breaking down the backup selection list to find the slow points and it appears its the data the data files stored on the locally connected HP SAN (currently 1.5 TB) which took around 20 hours on a full backup and also the exchange database (currently about 80 GB) which took around 23 hours on a full backup. All other resources on the selection list (system states, shadow copy components, shared folders, etc completed quickly in a few hours in total). So if the problem is with my users data and exchange database - what can I do to improve these? I have considered excluding multimedia files but not tried this yet. As mentioned earlier we are implementing quotas but this is a slow process as staff change here. As for the exchange database I am not too sure as my knowledge on exchange isn't extensive. I am aware the it does maintained every night on schedule and inbox's are restricted to 2GB per user.

 

I think in the long run I would like to replace the backup solution with some sort of snapshot backup as the network is in use nearly 24/7. However, if anyone could offer any suggestions for troubleshooting the data and exchange db elements I would greatly appreciate it. Let me know if I have been unclear above too.

 

Thanks

[6Foot2]

We are having similar issues with our backup [Full backup and verify: 3 days and 8 tapes: Tape: LTO3]

 

I had thought that the tapes had aged beyond their useful life so I replaced all the tapes in the library with no noticeable benefit [but perhaps you could look at your tapes to see if it would help you by refreshing your tape stock?]

 

We are virtualising our servers in the summer and so it is not worthwhile trying to obtain a new backup solution now: We will link on with what we have and coax te current solution to work until it is finally retired.

[lafleur1977]

We are having similar issues with our backup [Full backup and verify: 3 days and 8 tapes: Tape: LTO3]

 

I had thought that the tapes had aged beyond their useful life so I replaced all the tapes in the library with no noticeable benefit [but perhaps you could look at your tapes to see if it would help you by refreshing your tape stock?]

 

We are virtualising our servers in the summer and so it is not worthwhile trying to obtain a new backup solution now: We will link on with what we have and coax te current solution to work until it is finally retired.

 

Thanks for your reply. We backup to disc first and then replicate to tape. The disc aspect seems to be the problem rather than tape, which replicate in a few hours.

[DMcCoy]

How many files and what AV on backup/servers?

[lafleur1977]

How many files and what AV on backup/servers?

 

For data...

Files: 1,624,661

Directories: 215,658

 

We use Sophos Anti-Virus on all of our servers, including our media server, which is provided by our LA.

 

Thanks

[Duke5A]

I just went through this a couple months ago and the problem turned out to be something stupid. We have both disk and tape backup jobs and it was only the disk that was jamming up; jobs would take four times as long as they should. The problem was one of the two cache batteries on the RAID controller went dead and of the mountains of HP monitoring software loaded on the thing nothing even so much as squawked. We only found it when running diagnostics by hand. Even the pullout LED diagnostic indicator on the server was all lit green. When I pulled the RAID card out it was more than evident as what battery was fried since it was bulged out to the point I had problems removing it. After swapping in a new battery write-ahead cache was re-enabled and speeds picked up considerably.

[teejay]

Try doing a test backup straight to tape rather than to disk. If that is still slow, then check you have the latest client on the target server and also try disabling antivirus on the target server and also the backup server, also check your network configuration and nic drivers. If straight to tape is fine, then check you haven't got indexing turned on on the disk volume, also test the speed you can write to the disk by using some benchmarking software or just by copying a large file onto it.

[DMcCoy]

For data...

Files: 1,624,661

Directories: 215,658

 

We use Sophos Anti-Virus on all of our servers, including our media server, which is provided by our LA.

 

Thanks

 

Is sophos set to read/write scan or both?

 

I've seen sophos double the backup time with large numbers of files from a remote agent.

[old_n07]

BE was slower backing up to disc than to the tape library for us

[Cache]

I would say it's probably down to the amount of data and I don't know whether there is a lot you can do about it.

 

You could check and see if it is using software compression rather then hardware/none if you don't need to compress the data for space reasons.

 

Another point, the differential backups you mention will be backing up data that has changed since the last full backup, so:

• on Friday 1 you do your full backup
  
• Monday 1 will be data that changed saturday, sunday & monday
  
• Tuesday 1 will be data that changed saturday, sunday, monday & tuesday
  
• ...
  
• so that when you get to Thursday 4 you are backing up all changes that have occured over the last 3 weeks
  

 

So it is potentially a large amount of data that is being backed up by the time you get to the 3rd/4th week. We don't have anywhere near the same amount of data as you but we do a full backup on Friday and a Diff Mon - Thurs on the last physical box we've got. Depending on how many of your differential backups are overlapping, this might relieve this issue if you can do a more regular full backup.

 

Another thing to check would be the size of your backup2disk files to see what size it is. It could be the large number of files are fragmenting the system so much (but would be suprised since it's on a SAN).

 

I haven't used backupexec in a B2D environment for a while so it could be my ideas are way off, but still worth a look.

[lafleur1977]

Is sophos set to read/write scan or both?

 

I've seen sophos double the backup time with large numbers of files from a remote agent.

 

Not sure off the top of my head but I will check tomorrow. When you mentioned AV in your last reply I checked the AV on the media server and the daily scan starts at 9pm daily and doesn't seem to finish until 9am! I also noticed a duplicate scheduled task to do an AV scan on most of the data on the HP SAN. I will look into this more tomorrow and try disabling AV. Might have been a very good suggestion that

[DMcCoy]

iometer is also worth a try to find out sustained read/write speed from the drives in the backup server. I used to have sophos stopped and started via the pre/post commands in BE.

[lafleur1977]

I would say it's probably down to the amount of data and I don't know whether there is a lot you can do about it.

 

You could check and see if it is using software compression rather then hardware/none if you don't need to compress the data for space reasons.

 

Another point, the differential backups you mention will be backing up data that has changed since the last full backup, so:

• on Friday 1 you do your full backup
  
• Monday 1 will be data that changed saturday, sunday & monday
  
• Tuesday 1 will be data that changed saturday, sunday, monday & tuesday
  
• ...
  
• so that when you get to Thursday 4 you are backing up all changes that have occured over the last 3 weeks
  

 

So it is potentially a large amount of data that is being backed up by the time you get to the 3rd/4th week. We don't have anywhere near the same amount of data as you but we do a full backup on Friday and a Diff Mon - Thurs on the last physical box we've got. Depending on how many of your differential backups are overlapping, this might relieve this issue if you can do a more regular full backup.

 

Another thing to check would be the size of your backup2disk files to see what size it is. It could be the large number of files are fragmenting the system so much (but would be suprised since it's on a SAN).

 

I haven't used backupexec in a B2D environment for a while so it could be my ideas are way off, but still worth a look.

 

I have thought the amount of data would be the problem since I first started - trouble is introducing quotas when staff are use to having "infinite" storage has been difficult so far. We are a high school with 1400 users - What kind of size in your user base and whats your backup size?

 

Never thought about changing the compression options but will try that suggestion out. That's correct in how you describe the differential backups. They usually start at about 100 GB per diff on the first Monday and grow to about 200 GB before the next Full backup. Problem is all the diff backups throughout the month take about the same 25 hours, so they are overlapping from the beginning. I have been trying to concentrate on improving the full backup first because there is so many different elements to the backup config.

 

I have also looked at the .bkf file sizes (originally set to 1GB). I increased these to 10, 20 and 50 GB but these didn't make alot of difference. I have left it at 10 GB currently. Like you say it should be okay being a SAN but I make sure to defragment regularly (every half term - 6-8 weeks) and it literally takes about 3 days!

 

Thanks for your suggestions

[Cache]

Give Contig a try to defragment the files and the disk, I've been told it does a much better job at defragmentation then the built in utilities of Windows. I haven't go all the command line switches to hand, think they are on my backup dvd at home so will take a look tonight for you. It can also defragment the freespace which did help matters slightly for us.

 

Contig

 

We're a much smaller site, so arround the 500 user mark. My backup capacity has only increased slightly over the past couple of years, so my full backups are arround the 800GB mark, but I suspect over the last 6 months that's down to Veeam deduplicating a lot of the data because I have added several more servers since virtualising.

 

I know your pain with the quota's though, that's my next project to revisit again.

 

It's odd that 200GB doesn't take any longer to backup then 100GB, 25 hours does seem very slow even for that!

Edited April 25, 2013 by Cache

[lafleur1977]

 

It's odd that 200GB doesn't take any longer to backup then 100GB, 25 hours does seem very slow even for that!

 

Looking at Tuesdays diff backup, it was just over 100 GB and it took 26 hours and 10 minutes. As each diff backup completes throughout the month, and the data increases the job rate also increases. Starts at around 70 MB/min and goes up to around 150MB/min by the end of the month.

 

I'm going to look into a few of the suggestions above and yours too anyway. We are also looking into Virtualisation as our servers will need replacing soon so it might be that we just make do until then. At least the backups are working and we can restore successfully

 

Thanks

[Jaydeep]

Have you confirmed if the antivirus is set to sn when data is being read. This could result in considerably slow backups. I don't think the bkf file size at its current 10 GB is a problem, it is OK to leave it at that.

 

The main concern when the backups are slow i the fragmentation both at source and the B2D location. I would also suggest you to monitor the disk IOs

 

You could refer to this document http://www.symantec.com/docs/DOC3276 and even if it says for 2010, follow it as most of the suggestions are not BE version specific but very generic. I have seen them fix most performance issues.

 

Thanks

Edited April 26, 2013 by Jaydeep

[lafleur1977]

Have you confirmed if the antivirus is set to sn when data is being read. This could result in considerably slow backups. I don't think the bkf file size at its current 10 GB is a problem, it is OK to leave it at that.

 

The main concern when the backups are slow i the fragmentation both at source and the B2D location. I would also suggest you to monitor the disk IOs

 

You could refer to this document http://www.symantec.com/docs/DOC3276 and even if it says for 2010, follow it as most of the suggestions are not BE version specific but very generic. I have seen them fix most performance issues.

 

Thanks

 

Yes I checked. All of our servers has a sophos policy and on access scanning is disabled. I have disabled anti-virus scanning on our servers and am just letting a few backups complete and see if that improves it.

 

Yeah fragmentation could be the issue but defragmenting the volumes takes days so I usually do this during closure periods. I am going to try contig and see of that works better next time.

 

Thanks for the link to the document, I will look through that too

[lafleur1977]

Have you confirmed if the antivirus is set to sn when data is being read. This could result in considerably slow backups. I don't think the bkf file size at its current 10 GB is a problem, it is OK to leave it at that.

 

The main concern when the backups are slow i the fragmentation both at source and the B2D location. I would also suggest you to monitor the disk IOs

 

You could refer to this document http://www.symantec.com/docs/DOC3276 and even if it says for 2010, follow it as most of the suggestions are not BE version specific but very generic. I have seen them fix most performance issues.

 

Thanks

 

Yes I checked. All of our servers has a sophos policy and on access scanning is disabled. I have disabled anti-virus scanning on our servers and am just letting a few backups complete and see if that improves it.

 

Yeah fragmentation could be the issue but defragmenting the volumes takes days so I usually do this during closure periods. I am going to try contig and see of that works better next time.

 

Thanks for the link to the document, I will look through that too

[Cache]

Can't find my old batch file using contig, but it was scheduled to run about once a month I think.

 

The command was something like contig -s -v E:\* where e was the backup2disk drive, -s makes it recurse subdirectories and -v made it verbose.

 

It will take a while to run and might require a few runs through as it will try to reduce fragmentation of files as much as possible depending on the freespace it has to locate files to.

 

You might not see immediate benefits with it either, depending on how far through you are replacing the old B2D files which were 1GB with the 10GB ones, I think it was about a month after as the freespace before the most benefit was recieved.

Edited April 26, 2013 by Cache