Trickle Backup

[Reaper]

My Network Manager is after a backup program that will do trickle backup over the internet from an external site to our main servers.

 

Any ideas of which software would be worth a try?

 

 

Thanks

[Jona]

The key search term here is going to be differential or incremental backup. AFAIK all the major backup players do product that will do this. I've had success with a combination of smartsync pro and ftp but not in a production environment.....

 

Cheers

Jona

[dhicks]

Any ideas of which software would be worth a try?

 

rsync.

 

--

David Hicks

[torledo]

My Network Manager is after a backup program that will do trickle backup over the internet from an external site to our main servers.

 

Any ideas of which software would be worth a try?

 

 

Thanks

 

Never hears that term 'trickle' used before to describe a backup process..

 

It's an apt description.

[Guest monkeyx]

Unison is also good, similar to Rsync.

[Reaper]

When he uses the term trickle I think he's referring to it being a slow process rather than a CPU intensive move. So as that way not to take up all the bandwidth, probably similar to WOWs update process that runs in the background.

 

he also mentioned stuff like Cybervault and Mozy

Edited April 11, 2008 by Reaper

[steve]

Have a look a Symantec - Backup Exec Continuous Data Protection

 

Only backs up changes to files, and runs all the time so bandwidth should be minimal - also allows bandwidth throttling.

[torledo]

When he uses the term trickle I think he's referring to it being a slow process rather than a CPU intensive move. So as that way not to take up all the bandwidth, probably similar to WOWs update process that runs in the background.

 

he also mentioned stuff like Cybervault and Mozy

 

If he's talkiing about a backup operation not overhelming a slow speed WAN link then the correct term for ensuring that doesn't happen is WAN optimization, specifically methods such as bandwidth throttling as mentioned by steve. I do like the idea of describing it as 'trickle' but it's not a known terminology.

[Joedetic]

It's not an appropriate method for you to use for this, but something I keep meaning to take a look at is for Linux and it's DRBD (Distributed replication block device or something like that). Another machine on the network acts as a local block device and you can get it to replicate all blocks on the hard drive to the DRBD box which then takes regular snapshots and saves those so it's a very comprehensive sytem of live backup. (Essentially mirroring with archiving)

 

In your situation I'd go with rsync. The initial sync will take a while over a WAN link though.

[dhicks]

It's not an appropriate method for you to use for this, but something I keep meaning to take a look at is for Linux and it's DRBD

 

Just got to having a look at this myself. I'm intending to use it with Xen - have writes to Xen virtual machine file systems mirrored to a separate physical machine for instant exploding-power-supply-proof failover. I'm having issues getting various bits to work together, though. Ubuntu 7.10 crashes on high I/O on our new (and very, very, cheap) Dell servers - something to do with glibc, I think. I'm using 7.04 instead. This runs Xen fine, but has issues with doing RAID 10 with mdadm - bizzare error necessitates reboot from install CD to fix broken system. And now that Xen is installed, the kernel header versions have changed so drdb won't install until I find linux-headers-2.6.19-4-server...

 

--

David Hicks

[Joedetic]

Yeah it all looks like it could be quite complicated for what I'd want it for anyway..that and I'd need to look at locating a machine at the other end of the site connected by fibre or something so as not to pull huge load on the network.

[dhicks]

Yeah it all looks like it could be quite complicated for what I'd want it for anyway..that and I'd need to look at locating a machine at the other end of the site connected by fibre or something so as not to pull huge load on the network.

 

It should be less complicated if you're just cloning a file system - no Xen running means no messing around with header files and installing DRDB should just be a case of using apt-get to install the appropriate package. According to the DRDB documentation (which is very comprehensive and well-written, by the way) you can limit the bandwidth taken up by the replication system (I guess each written block is marked as "dirty" and shoved in a queue to be written when bandwidth allows?). Depends on how much data you have written to your server, too - I bet you don't actually get that much written, most requests will be reads.

 

The way the DRDB documentation recommends linking two machines seems to be to simply plug them directly together via ethernet (I guess you could use a number of bonded ethernet ports). Skip the whole SAN business, you don't even need a switch to be able to have physically separated (well, okay, by up to 100m - technically wouldn't survive a nuclear bomb blast...) redundant-hardware systems capable of instant failover!

 

I'm going to try re-installing Ubuntu 7.04 and this time compiling both Xen and DRDB from source to ensure I stick with the correct kernel headers. If this proves too much messing around then I'm switching to using just LVM volumes with periodic snapshots being used for backups.

 

--

David Hicks

[Joedetic]

Well a company I had an interview a while ago were explaining about this being used for their backup process. They weren't using Xen but VMware on Gentoo. I had origionally wanted to learn to do this on FreeBSD but if it's going into the school where they'll have to support it when i'm gone i'm thinking maybe Debian/Ubuntu.

 

Again...it's something I need to play with in VMs before I start in June.

 

Out of interest (because I'm not entirely sure myself) what's the restore process? Just SCP the files back or something? But what if you've got to do an entire restore of a box?

[dhicks]

if it's going into the school where they'll have to support it when i'm gone i'm thinking maybe Debian/Ubuntu.

 

I'd definatly wait a couple of weeks for Ubuntu 8.04 - Xen on 7.10 seems to have stability problems and mdadm on 7.04 is rather screwy too. Might want to see how you get on with Debian instead, or maybe Ubuntu 6.06 as it seems there's a seperate build of DRDB available for it.

 

Out of interest (because I'm not entirely sure myself) what's the restore process? Just SCP the files back or something? But what if you've got to do an entire restore of a box?

 

What, how do you restore a virtual machine, you mean? The idea is that DRDB keeps two filesystems exactly in synch, so you two exact copies on different machines. If one machine goes down then the other can take over. If you can't get the first machine back up again with the data intact then yes, I guess you'd simply have to copy the data over and start again.

 

--

David Hicks

[Joedetic]

Well yeah, if you're using it to backup virtual machines...i was planning host OS backup...or doesn't it work like that?

[dhicks]

Well yeah, if you're using it to backup virtual machines...i was planning host OS backup...or doesn't it work like that?

 

Mount the host OS on a DRDB block device, you mean? Hmm, didn't think of that one - might work, try it and see what happens.

 

--

David Hicks

[stratisphere]

I'd highly recommend BackupPC: Open Source Backup to disk

 

BackupPC comes with a clever pooling scheme that minimizes disk storage and disk I/O. Identical files across multiple backups of the same or different PCs are stored only once resulting in substantial savings in disk storage and disk I/O.

 

Not only can it backup files tho, it can also restore them to exactually where they were. Also gives a version history, so you can look at a certain file, then get a history of which backups it was changed... good for finding out when files were deleted.

 

We have a server setup with it with a 1TB drive. Backups all the servers on our network. We have it set for weekly full backups, and nightly incremental backups, regardless of the server and it's role. After pooling & compression our pool is at 177gb for all 14 of our servers (which are a mix of windows server 2k3 and linux (debian, redhat & trixbox for VoIP)).

 

It can be a bit to get your head around at first, but there is a brill article over at Back Up Linux And Windows Systems With BackupPC | HowtoForge - Linux Howtos and Tutorials on how to get it setup.

 

Also, it has an archive feature. You can set it to mirror it's entire pool to another server... aka offsite backups. So it can be set to transfer the current pool. Ok 177gbs is a bit much. But we're lucky to have a 10mb fibre to our school. And the council have complained (yet).