Block level backup/storage/duplication

[j17sparky]

We're looking at ways to backup our SAN. We've got a nice little script which snapshots and copies our virtual machines and drops the machine image in a folder, so that part of it is fine. We just need a good way of storing the images without having to have 10s of TBs of diskspace.

Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage?

R1soft seem like they do what i need, but does anyone know of a free way of doing it?


Cheers


To sum it up if i wasnt clear, i want to; Copy the machine images to a server, that server then find all duplicate blocks and deal with them, reducing disk usage, while still allowing us full (non-differencial) images to pull straight out in teh event of disaster

[dhicks]

Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage?

If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication.

--
David Hicks

[j17sparky]

If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication.

--
David Hicks

Nice, Opensolaris here i come.

[apaton]

Nice, Opensolaris here i come.

Hold your horses fellas!

ZFS Dedupe is available only in the latest development source code only.
I'm running OpenSolaris build 127 which is the latest binary update and that doesn't have it available.

I'm sure it will be out in a week or two, but this is fresh new code so I wouldn't go production just yet!!!


Andy

[apaton]

ZFS Dedupe delayed due to bug in Fletcher-4 and hash collisions.

We now have to wait to build 129

With luck Santa will bring dedupe for Christmas.!

Andy

[apaton]

Well we didn't have to wait for OpenSolaris 129, they built 128a.

The following example is coping Xvnc 25 times in to a dedup and non-dedup enabled ZFS datasets.

apaton@vb-osol:~$ uname -a
SunOS vb-osol 5.11 snv_128a i86pc i386 i86pc Solaris

apaton@vb-osol:~$ pfexec mkfile -n 3g /disk1
apaton@vb-osol:~$ pfexec zpool create tank /disk1
apaton@vb-osol:~$ pfexec zfs create tank/track
apaton@vb-osol:~$ pfexec zfs set dedup=on tank/track
apaton@vb-osol:~$ pfexec zfs create tank/girl
apaton@vb-osol:~$ pfexec chmod 777 /tank/track /tank/girl
 
apaton@vb-osol:~$ cd /tank/track
apaton@vb-osol:/tank/track$ for i in 1 2 3 4 5  ; do mkdir $i; for p in 1 2 3 4 5  ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done
apaton@vb-osol:/tank/track$ cd /tank/girl
 apaton@vb-osol:/tank/girl$ for i in 1 2 3 4 5  ; do mkdir $i; for p in 1 2 3 4 5  ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done

apaton@vb-osol:/tank/girl$ df -h | egrep "(^File|^tank)"
Filesystem            Size  Used Avail Use% Mounted on
tank                  2.9G   24K  2.9G   1% /tank
tank/girl             3.0G   85M  2.9G   3% /tank/girl
tank/track            3.0G   85M  2.9G   3% /tank/track
apaton@vb-osol:/tank/girl$ zfs list -r tank
NAME         USED  AVAIL  REFER  MOUNTPOINT
tank         169M  2.85G    24K  /tank
tank/girl   84.5M  2.85G  84.5M  /tank/girl
tank/track  84.5M  2.85G  84.5M  /tank/track
apaton@vb-osol:/tank/girl$ zpool list tank
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank   2.98G  88.2M  2.90G     2%  25.00x  ONLINE  -

I copied Xvnc 155 times then noticed not only is dedup working perfectly but my filesystem /tank/track grew!! Are we using 16% of the 3G ZFS pool?

This might be misleading in you use df in your filesystems monitoring scripts.

apaton@vb-osol:/tank/track$ find . -name Xvnc\* | wc -l
155

apaton@vb-osol:/tank/track$ zpool list tank
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank   2.98G  88.7M  2.90G     2%  155.00x  ONLINE  -

apaton@vb-osol:/tank/track$ df -h | egrep "(^File|^tank)"
Filesystem            Size  Used Avail Use% Mounted on
tank                  2.9G   24K  2.9G   1% /tank
tank/girl             3.0G   85M  2.9G   3% /tank/girl
tank/track            3.4G  524M  2.9G  16% /tank/track

apaton@vb-osol:/tank/track$ zfs list -r tank
NAME         USED  AVAIL  REFER  MOUNTPOINT
tank         608M  2.84G    24K  /tank
tank/girl   84.5M  2.84G  84.5M  /tank/girl
tank/track   524M  2.84G   524M  /tank/track

[j17sparky]

Well we didn't have to wait for OpenSolaris 129, they built 128a.

you know what im doing tomorrow then