Trickle Backup

[Reaper]

My Network Manager is after a backup program that will do trickle backup over the internet from an external site to our main servers.

Any ideas of which software would be worth a try?


Thanks

[Jona]

The key search term here is going to be differential or incremental backup. AFAIK all the major backup players do product that will do this. I've had success with a combination of smartsync pro and ftp but not in a production environment.....

Cheers
Jona

[dhicks]

Any ideas of which software would be worth a try?

rsync.

--
David Hicks

[torledo]

My Network Manager is after a backup program that will do trickle backup over the internet from an external site to our main servers.

Any ideas of which software would be worth a try?


Thanks

Never hears that term 'trickle' used before to describe a backup process..

It's an apt description.

[monkeyx]

Unison is also good, similar to Rsync.

[Reaper]

When he uses the term trickle I think he's referring to it being a slow process rather than a CPU intensive move. So as that way not to take up all the bandwidth, probably similar to WOWs update process that runs in the background.

he also mentioned stuff like Cybervault and Mozy

[steve]

Have a look a Symantec - Backup Exec Continuous Data Protection

Only backs up changes to files, and runs all the time so bandwidth should be minimal - also allows bandwidth throttling.

[torledo]

When he uses the term trickle I think he's referring to it being a slow process rather than a CPU intensive move. So as that way not to take up all the bandwidth, probably similar to WOWs update process that runs in the background.

he also mentioned stuff like Cybervault and Mozy

If he's talkiing about a backup operation not overhelming a slow speed WAN link then the correct term for ensuring that doesn't happen is WAN optimization, specifically methods such as bandwidth throttling as mentioned by steve. I do like the idea of describing it as 'trickle' but it's not a known terminology.

[Joedetic]

It's not an appropriate method for you to use for this, but something I keep meaning to take a look at is for Linux and it's DRBD (Distributed replication block device or something like that). Another machine on the network acts as a local block device and you can get it to replicate all blocks on the hard drive to the DRBD box which then takes regular snapshots and saves those so it's a very comprehensive sytem of live backup. (Essentially mirroring with archiving)

In your situation I'd go with rsync. The initial sync will take a while over a WAN link though.

[dhicks]

It's not an appropriate method for you to use for this, but something I keep meaning to take a look at is for Linux and it's DRBD

Just got to having a look at this myself. I'm intending to use it with Xen - have writes to Xen virtual machine file systems mirrored to a separate physical machine for instant exploding-power-supply-proof failover. I'm having issues getting various bits to work together, though. Ubuntu 7.10 crashes on high I/O on our new (and very, very, cheap) Dell servers - something to do with glibc, I think. I'm using 7.04 instead. This runs Xen fine, but has issues with doing RAID 10 with mdadm - bizzare error necessitates reboot from install CD to fix broken system. And now that Xen is installed, the kernel header versions have changed so drdb won't install until I find linux-headers-2.6.19-4-server...

--
David Hicks

[Joedetic]

Yeah it all looks like it could be quite complicated for what I'd want it for anyway..that and I'd need to look at locating a machine at the other end of the site connected by fibre or something so as not to pull huge load on the network.

[dhicks]

Yeah it all looks like it could be quite complicated for what I'd want it for anyway..that and I'd need to look at locating a machine at the other end of the site connected by fibre or something so as not to pull huge load on the network.

It should be less complicated if you're just cloning a file system - no Xen running means no messing around with header files and installing DRDB should just be a case of using apt-get to install the appropriate package. According to the DRDB documentation (which is very comprehensive and well-written, by the way) you can limit the bandwidth taken up by the replication system (I guess each written block is marked as "dirty" and shoved in a queue to be written when bandwidth allows?). Depends on how much data you have written to your server, too - I bet you don't actually get that much written, most requests will be reads.

The way the DRDB documentation recommends linking two machines seems to be to simply plug them directly together via ethernet (I guess you could use a number of bonded ethernet ports). Skip the whole SAN business, you don't even need a switch to be able to have physically separated (well, okay, by up to 100m - technically wouldn't survive a nuclear bomb blast...) redundant-hardware systems capable of instant failover!

I'm going to try re-installing Ubuntu 7.04 and this time compiling both Xen and DRDB from source to ensure I stick with the correct kernel headers. If this proves too much messing around then I'm switching to using just LVM volumes with periodic snapshots being used for backups.

--
David Hicks

[Joedetic]

Well a company I had an interview a while ago were explaining about this being used for their backup process. They weren't using Xen but VMware on Gentoo. I had origionally wanted to learn to do this on FreeBSD but if it's going into the school where they'll have to support it when i'm gone i'm thinking maybe Debian/Ubuntu.

Again...it's something I need to play with in VMs before I start in June.

Out of interest (because I'm not entirely sure myself) what's the restore process? Just SCP the files back or something? But what if you've got to do an entire restore of a box?

[dhicks]

if it's going into the school where they'll have to support it when i'm gone i'm thinking maybe Debian/Ubuntu.

I'd definatly wait a couple of weeks for Ubuntu 8.04 - Xen on 7.10 seems to have stability problems and mdadm on 7.04 is rather screwy too. Might want to see how you get on with Debian instead, or maybe Ubuntu 6.06 as it seems there's a seperate build of DRDB available for it.

Out of interest (because I'm not entirely sure myself) what's the restore process? Just SCP the files back or something? But what if you've got to do an entire restore of a box?

What, how do you restore a virtual machine, you mean? The idea is that DRDB keeps two filesystems exactly in synch, so you two exact copies on different machines. If one machine goes down then the other can take over. If you can't get the first machine back up again with the data intact then yes, I guess you'd simply have to copy the data over and start again.

--
David Hicks

[Joedetic]

Well yeah, if you're using it to backup virtual machines...i was planning host OS backup...or doesn't it work like that?