Guest Guest Posted November 13, 2009 Posted November 13, 2009 We're looking at ways to backup our SAN. We've got a nice little script which snapshots and copies our virtual machines and drops the machine image in a folder, so that part of it is fine. We just need a good way of storing the images without having to have 10s of TBs of diskspace. Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage? R1soft seem like they do what i need, but does anyone know of a free way of doing it? Cheers To sum it up if i wasnt clear, i want to; Copy the machine images to a server, that server then find all duplicate blocks and deal with them, reducing disk usage, while still allowing us full (non-differencial) images to pull straight out in teh event of disaster
dhicks Posted November 13, 2009 Posted November 13, 2009 Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage? If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication. -- David Hicks
Guest Guest Posted November 13, 2009 Posted November 13, 2009 If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication. -- David Hicks Nice, Opensolaris here i come.
apaton Posted November 15, 2009 Posted November 15, 2009 Nice, Opensolaris here i come. Hold your horses fellas! ZFS Dedupe is available only in the latest development source code only. I'm running OpenSolaris build 127 which is the latest binary update and that doesn't have it available. I'm sure it will be out in a week or two, but this is fresh new code so I wouldn't go production just yet!!! Andy
apaton Posted November 25, 2009 Posted November 25, 2009 ZFS Dedupe delayed due to bug in Fletcher-4 and hash collisions. We now have to wait to build 129 With luck Santa will bring dedupe for Christmas.! Andy
apaton Posted December 6, 2009 Posted December 6, 2009 (edited) Well we didn't have to wait for OpenSolaris 129, they built 128a. The following example is coping Xvnc 25 times in to a dedup and non-dedup enabled ZFS datasets. [font=Courier New]apaton@vb-osol:~$ uname -a SunOS vb-osol 5.11 [b]snv_128a[/b] i86pc i386 i86pc Solaris apaton@vb-osol:~$ pfexec mkfile -n 3g /disk1 apaton@vb-osol:~$ pfexec zpool create tank /disk1 apaton@vb-osol:~$ pfexec zfs create tank/track apaton@vb-osol:~$ pfexec zfs set dedup=on tank/track apaton@vb-osol:~$ pfexec zfs create tank/girl apaton@vb-osol:~$ pfexec chmod 777 /tank/track /tank/girl [/font][font=Courier New] apaton@vb-osol:~$ cd /tank/track[/font] [font=Courier New]apaton@vb-osol:/tank/track$ for i in 1 2 3 4 5 ; do mkdir $i; for p in 1 2 3 4 5 ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done [/font][font=Courier New]apaton@vb-osol:/tank/track$[/font][font=Courier New] cd /tank/girl[/font] [font=Courier New] apaton@vb-osol:/tank/girl$ for i in 1 2 3 4 5 ; do mkdir $i; for p in 1 2 3 4 5 ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done apaton@vb-osol:/tank/girl$ df -h | egrep "(^File|^tank)" Filesystem Size Used Avail Use% Mounted on tank 2.9G 24K 2.9G 1% /tank tank/girl 3.0G 85M 2.9G 3% /tank/girl tank/track 3.0G 85M 2.9G 3% /tank/track apaton@vb-osol:/tank/girl$ zfs list -r tank NAME USED AVAIL REFER MOUNTPOINT tank 169M 2.85G 24K /tank tank/girl 84.5M 2.85G 84.5M /tank/girl tank/track 84.5M 2.85G 84.5M /tank/track apaton@vb-osol:/tank/girl$ zpool list tank NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 2.98G 88.2M 2.90G 2% [b]25.00x[/b] ONLINE - [/font] I copied Xvnc 155 times then noticed not only is dedup working perfectly but my filesystem /tank/track grew!! Are we using 16% of the 3G ZFS pool? This might be misleading in you use df in your filesystems monitoring scripts. [font=Courier New]apaton@vb-osol:/tank/track$ find . -name Xvnc\* | wc -l 155 apaton@vb-osol:/tank/track$ zpool list tank NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 2.98G 88.7M 2.90G 2% [b]155.00x[/b] ONLINE - apaton@vb-osol:/tank/track$ df -h | egrep "(^File|^tank)" Filesystem Size Used Avail Use% Mounted on tank 2.9G 24K 2.9G 1% /tank tank/girl 3.0G 85M 2.9G 3% /tank/girl tank/track [b]3.4G[/b] 524M 2.9G [b]16%[/b] /tank/track apaton@vb-osol:/tank/track$ zfs list -r tank NAME USED AVAIL REFER MOUNTPOINT tank 608M 2.84G 24K /tank tank/girl 84.5M 2.84G 84.5M /tank/girl tank/track 524M 2.84G 524M /tank/track[/font] Edited December 7, 2009 by apaton typo
Guest Guest Posted December 7, 2009 Posted December 7, 2009 Well we didn't have to wait for OpenSolaris 129, they built 128a. you know what im doing tomorrow then
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now