+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 19
Enterprise Software Thread, BackupExec 12.5- full backups 50 hours! in Technical; Hello, I have posted about this before but am still no closer to a solution. I have inherited a backup ...
  1. #1
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11

    BackupExec 12.5- full backups 50 hours!

    Hello,

    I have posted about this before but am still no closer to a solution.

    I have inherited a backup solution which uses BackupExec 12.5 and I am having trouble with extremely slow backup times. Currently full backups are taking 50 hours including verify and differentlial backups are taking about 25 hours. The backup time has only increased despite trying to decrease the amount of data by implementing quotas (still in progress however) and I could do with a little help to try and get the backup times down to a more realistic time.

    Firstly I shall explain the solution:

    Software: BackupExec 12.5 media server running on Windows Server 2008 64-bit. This server is also the main file server.
    Hardware: BackupExec 12.5 Media/File server -
    Spec: Intel Xeon E5310 1.60 GHz, 8GB RAM and 1 Gbps NIC.
    Storage: HP SAN configured by a hardware RAID and formatted as one 5 ish TB NTFS partition. This is used for the main data storage.
    HP Disk Array configured by a Windows software RAID (as it wouldn't work in a hardware RAID) and formatted as one 5 ish TB NTFS partition. This is used for backup storage.
    HP Tape LT03 drive.

    BackupExec 12.5 software configuration: Media server has been configured with one daily backup selection list. This currently includes the data files stored on the locally connected HP SAN (currently 1.5 TB), system states for two DC's, system state of an exchange server, exchange database (currently about 80 GB), system state of print server, shadow copy components of print server (not sure what this does and if its needed?), and a few shared folders on a couple of other servers.

    Two folders have been created on the HP disc array. One called Full and the other Differentials. Both of these have corresponding backup to disc folders under devices. Both were are configured with over right protection of 27 days but recently I have had to decrease these to 3 weeks and 1 week respectively due to low disc space, but I know this can be rectified by increasing the storage.

    Backup jobs have been created and use the selection list I described above:

    Mon, Tues, Weds, Thurs Differential daily backups to the Differential folder.
    Fri 1 Full monthly backup to the Full folder. These is then duplicated to tape.
    Fri 2, fri 3 and fri 4 Differential weekly backups to the Differential folder. These are then duplicated to tape.

    Conclusion:

    So the server is configured to do daily differential backups and a monthly full backup. I believe this is suppose to be the fastest method as diffs only backup the changes made since the last full. Full backups are currently around 1.7 TB's and is taking nearly 50 hours to complete, including verify. Diff backups are currently between 100 - 200 GB and are taking about 25 hours to complete, including verify. This also means our daily backups are now overlapping. Duplication to tape is around a few hours and isn't a problem. One thing that I am confused by is the job rate which after checking the job logs varies for each resource. Overall for full backups its around 700MB/min and for diffs is between 70 - 150 MB/min. Backups generally do complete successfully but the backup to disc completion time has done nothing but increase since I started here. I have tried one suggestion from a previous post to increase the .bkf file size as these were set to 1GB but I have increased them up to 50 GB and this only made a minor improvement. I have also tried breaking down the backup selection list to find the slow points and it appears its the data the data files stored on the locally connected HP SAN (currently 1.5 TB) which took around 20 hours on a full backup and also the exchange database (currently about 80 GB) which took around 23 hours on a full backup. All other resources on the selection list (system states, shadow copy components, shared folders, etc completed quickly in a few hours in total). So if the problem is with my users data and exchange database - what can I do to improve these? I have considered excluding multimedia files but not tried this yet. As mentioned earlier we are implementing quotas but this is a slow process as staff change here. As for the exchange database I am not too sure as my knowledge on exchange isn't extensive. I am aware the it does maintained every night on schedule and inbox's are restricted to 2GB per user.

    I think in the long run I would like to replace the backup solution with some sort of snapshot backup as the network is in use nearly 24/7. However, if anyone could offer any suggestions for troubleshooting the data and exchange db elements I would greatly appreciate it. Let me know if I have been unclear above too.

    Thanks

  2. #2

    DaveP's Avatar
    Join Date
    Oct 2006
    Location
    Can't talk now: The mother-ship is calling!
    Posts
    8,982
    Thank Post
    352
    Thanked 1,300 Times in 890 Posts
    Blog Entries
    4
    Rep Power
    1131
    We are having similar issues with our backup [Full backup and verify: 3 days and 8 tapes: Tape: LTO3]

    I had thought that the tapes had aged beyond their useful life so I replaced all the tapes in the library with no noticeable benefit [but perhaps you could look at your tapes to see if it would help you by refreshing your tape stock?]

    We are virtualising our servers in the summer and so it is not worthwhile trying to obtain a new backup solution now: We will link on with what we have and coax te current solution to work until it is finally retired.

  3. Thanks to DaveP from:

    lafleur1977 (24th April 2013)

  4. #3
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11
    Quote Originally Posted by DaveP View Post
    We are having similar issues with our backup [Full backup and verify: 3 days and 8 tapes: Tape: LTO3]

    I had thought that the tapes had aged beyond their useful life so I replaced all the tapes in the library with no noticeable benefit [but perhaps you could look at your tapes to see if it would help you by refreshing your tape stock?]

    We are virtualising our servers in the summer and so it is not worthwhile trying to obtain a new backup solution now: We will link on with what we have and coax te current solution to work until it is finally retired.
    Thanks for your reply. We backup to disc first and then replicate to tape. The disc aspect seems to be the problem rather than tape, which replicate in a few hours.

  5. #4
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,464
    Thank Post
    10
    Thanked 496 Times in 436 Posts
    Rep Power
    113
    How many files and what AV on backup/servers?

  6. Thanks to DMcCoy from:

    lafleur1977 (24th April 2013)

  7. #5
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11
    Quote Originally Posted by DMcCoy View Post
    How many files and what AV on backup/servers?
    For data...
    Files: 1,624,661
    Directories: 215,658

    We use Sophos Anti-Virus on all of our servers, including our media server, which is provided by our LA.

    Thanks

  8. #6
    Duke5A's Avatar
    Join Date
    Jul 2010
    Posts
    807
    Thank Post
    83
    Thanked 132 Times in 115 Posts
    Blog Entries
    8
    Rep Power
    32
    I just went through this a couple months ago and the problem turned out to be something stupid. We have both disk and tape backup jobs and it was only the disk that was jamming up; jobs would take four times as long as they should. The problem was one of the two cache batteries on the RAID controller went dead and of the mountains of HP monitoring software loaded on the thing nothing even so much as squawked. We only found it when running diagnostics by hand. Even the pullout LED diagnostic indicator on the server was all lit green. When I pulled the RAID card out it was more than evident as what battery was fried since it was bulged out to the point I had problems removing it. After swapping in a new battery write-ahead cache was re-enabled and speeds picked up considerably.

  9. Thanks to Duke5A from:

    lafleur1977 (24th April 2013)

  10. #7

    teejay's Avatar
    Join Date
    Apr 2008
    Posts
    3,187
    Thank Post
    285
    Thanked 775 Times in 585 Posts
    Rep Power
    336
    Try doing a test backup straight to tape rather than to disk. If that is still slow, then check you have the latest client on the target server and also try disabling antivirus on the target server and also the backup server, also check your network configuration and nic drivers. If straight to tape is fine, then check you haven't got indexing turned on on the disk volume, also test the speed you can write to the disk by using some benchmarking software or just by copying a large file onto it.

  11. Thanks to teejay from:

    lafleur1977 (24th April 2013)

  12. #8
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,464
    Thank Post
    10
    Thanked 496 Times in 436 Posts
    Rep Power
    113
    Quote Originally Posted by lafleur1977 View Post
    For data...
    Files: 1,624,661
    Directories: 215,658

    We use Sophos Anti-Virus on all of our servers, including our media server, which is provided by our LA.

    Thanks
    Is sophos set to read/write scan or both?

    I've seen sophos double the backup time with large numbers of files from a remote agent.

  13. Thanks to DMcCoy from:

    lafleur1977 (24th April 2013)

  14. #9
    old_n07's Avatar
    Join Date
    Jun 2012
    Location
    North Staffordshire
    Posts
    99
    Thank Post
    10
    Thanked 17 Times in 15 Posts
    Rep Power
    7
    BE was slower backing up to disc than to the tape library for us

  15. Thanks to old_n07 from:

    lafleur1977 (24th April 2013)

  16. #10
    Cache's Avatar
    Join Date
    Apr 2008
    Location
    Cumbria
    Posts
    1,225
    Thank Post
    455
    Thanked 177 Times in 174 Posts
    Blog Entries
    3
    Rep Power
    65
    I would say it's probably down to the amount of data and I don't know whether there is a lot you can do about it.

    You could check and see if it is using software compression rather then hardware/none if you don't need to compress the data for space reasons.

    Another point, the differential backups you mention will be backing up data that has changed since the last full backup, so:
    • on Friday 1 you do your full backup
    • Monday 1 will be data that changed saturday, sunday & monday
    • Tuesday 1 will be data that changed saturday, sunday, monday & tuesday
    • ...
    • so that when you get to Thursday 4 you are backing up all changes that have occured over the last 3 weeks


    So it is potentially a large amount of data that is being backed up by the time you get to the 3rd/4th week. We don't have anywhere near the same amount of data as you but we do a full backup on Friday and a Diff Mon - Thurs on the last physical box we've got. Depending on how many of your differential backups are overlapping, this might relieve this issue if you can do a more regular full backup.

    Another thing to check would be the size of your backup2disk files to see what size it is. It could be the large number of files are fragmenting the system so much (but would be suprised since it's on a SAN).

    I haven't used backupexec in a B2D environment for a while so it could be my ideas are way off, but still worth a look.

  17. Thanks to Cache from:

    lafleur1977 (24th April 2013)

  18. #11
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11
    Quote Originally Posted by DMcCoy View Post
    Is sophos set to read/write scan or both?

    I've seen sophos double the backup time with large numbers of files from a remote agent.
    Not sure off the top of my head but I will check tomorrow. When you mentioned AV in your last reply I checked the AV on the media server and the daily scan starts at 9pm daily and doesn't seem to finish until 9am! I also noticed a duplicate scheduled task to do an AV scan on most of the data on the HP SAN. I will look into this more tomorrow and try disabling AV. Might have been a very good suggestion that

  19. #12
    DMcCoy's Avatar
    Join Date
    Oct 2005
    Location
    Isle of Wight
    Posts
    3,464
    Thank Post
    10
    Thanked 496 Times in 436 Posts
    Rep Power
    113
    iometer is also worth a try to find out sustained read/write speed from the drives in the backup server. I used to have sophos stopped and started via the pre/post commands in BE.

  20. Thanks to DMcCoy from:

    lafleur1977 (24th April 2013)

  21. #13
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11
    Quote Originally Posted by Cache View Post
    I would say it's probably down to the amount of data and I don't know whether there is a lot you can do about it.

    You could check and see if it is using software compression rather then hardware/none if you don't need to compress the data for space reasons.

    Another point, the differential backups you mention will be backing up data that has changed since the last full backup, so:
    • on Friday 1 you do your full backup
    • Monday 1 will be data that changed saturday, sunday & monday
    • Tuesday 1 will be data that changed saturday, sunday, monday & tuesday
    • ...
    • so that when you get to Thursday 4 you are backing up all changes that have occured over the last 3 weeks


    So it is potentially a large amount of data that is being backed up by the time you get to the 3rd/4th week. We don't have anywhere near the same amount of data as you but we do a full backup on Friday and a Diff Mon - Thurs on the last physical box we've got. Depending on how many of your differential backups are overlapping, this might relieve this issue if you can do a more regular full backup.

    Another thing to check would be the size of your backup2disk files to see what size it is. It could be the large number of files are fragmenting the system so much (but would be suprised since it's on a SAN).

    I haven't used backupexec in a B2D environment for a while so it could be my ideas are way off, but still worth a look.
    I have thought the amount of data would be the problem since I first started - trouble is introducing quotas when staff are use to having "infinite" storage has been difficult so far. We are a high school with 1400 users - What kind of size in your user base and whats your backup size?

    Never thought about changing the compression options but will try that suggestion out. That's correct in how you describe the differential backups. They usually start at about 100 GB per diff on the first Monday and grow to about 200 GB before the next Full backup. Problem is all the diff backups throughout the month take about the same 25 hours, so they are overlapping from the beginning. I have been trying to concentrate on improving the full backup first because there is so many different elements to the backup config.

    I have also looked at the .bkf file sizes (originally set to 1GB). I increased these to 10, 20 and 50 GB but these didn't make alot of difference. I have left it at 10 GB currently. Like you say it should be okay being a SAN but I make sure to defragment regularly (every half term - 6-8 weeks) and it literally takes about 3 days!

    Thanks for your suggestions

  22. #14
    Cache's Avatar
    Join Date
    Apr 2008
    Location
    Cumbria
    Posts
    1,225
    Thank Post
    455
    Thanked 177 Times in 174 Posts
    Blog Entries
    3
    Rep Power
    65
    Give Contig a try to defragment the files and the disk, I've been told it does a much better job at defragmentation then the built in utilities of Windows. I haven't go all the command line switches to hand, think they are on my backup dvd at home so will take a look tonight for you. It can also defragment the freespace which did help matters slightly for us.

    Contig

    We're a much smaller site, so arround the 500 user mark. My backup capacity has only increased slightly over the past couple of years, so my full backups are arround the 800GB mark, but I suspect over the last 6 months that's down to Veeam deduplicating a lot of the data because I have added several more servers since virtualising.

    I know your pain with the quota's though, that's my next project to revisit again.

    It's odd that 200GB doesn't take any longer to backup then 100GB, 25 hours does seem very slow even for that!
    Last edited by Cache; 25th April 2013 at 09:45 AM.

  23. Thanks to Cache from:

    lafleur1977 (25th April 2013)

  24. #15
    lafleur1977's Avatar
    Join Date
    May 2009
    Location
    Lancashire
    Posts
    160
    Thank Post
    119
    Thanked 4 Times in 4 Posts
    Rep Power
    11
    Quote Originally Posted by Cache View Post

    It's odd that 200GB doesn't take any longer to backup then 100GB, 25 hours does seem very slow even for that!
    Looking at Tuesdays diff backup, it was just over 100 GB and it took 26 hours and 10 minutes. As each diff backup completes throughout the month, and the data increases the job rate also increases. Starts at around 70 MB/min and goes up to around 150MB/min by the end of the month.

    I'm going to look into a few of the suggestions above and yours too anyway. We are also looking into Virtualisation as our servers will need replacing soon so it might be that we just make do until then. At least the backups are working and we can restore successfully

    Thanks

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. BackupExec 12.5 backup solution - SLOW!
    By lafleur1977 in forum Enterprise Software
    Replies: 10
    Last Post: 25th January 2013, 11:16 AM
  2. Replies: 4
    Last Post: 11th August 2011, 03:39 PM
  3. VSS backup options and multiple backups - copy or full backup?
    By ianh64 in forum Windows Server 2008 R2
    Replies: 4
    Last Post: 21st January 2011, 02:00 PM
  4. Replies: 4
    Last Post: 26th July 2010, 02:23 PM
  5. Trying to get BackupExec 12.5 remote agent to install
    By dgrams in forum Windows Server 2008
    Replies: 4
    Last Post: 3rd February 2010, 08:33 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •