+ Post New Thread
Results 1 to 7 of 7
*nix Thread, Block level backup/storage/duplication in Technical; We're looking at ways to backup our SAN. We've got a nice little script which snapshots and copies our virtual ...
  1. #1


    Join Date
    Oct 2006
    Posts
    3,414
    Thank Post
    184
    Thanked 356 Times in 285 Posts
    Rep Power
    149

    Block level backup/storage/duplication

    We're looking at ways to backup our SAN. We've got a nice little script which snapshots and copies our virtual machines and drops the machine image in a folder, so that part of it is fine. We just need a good way of storing the images without having to have 10s of TBs of diskspace.

    Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage?

    R1soft seem like they do what i need, but does anyone know of a free way of doing it?


    Cheers


    To sum it up if i wasnt clear, i want to; Copy the machine images to a server, that server then find all duplicate blocks and deal with them, reducing disk usage, while still allowing us full (non-differencial) images to pull straight out in teh event of disaster

  2. #2

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,772
    Thank Post
    1,308
    Thanked 804 Times in 698 Posts
    Rep Power
    247
    Quote Originally Posted by j17sparky View Post
    Block level duplication (like windows SIS/VSS) seems the way to go. Can anyone point me in the right direction for linux based storage?
    If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication.

    --
    David Hicks

  3. #3


    Join Date
    Oct 2006
    Posts
    3,414
    Thank Post
    184
    Thanked 356 Times in 285 Posts
    Rep Power
    149
    Quote Originally Posted by dhicks View Post
    If I understand correctly, the ZFS file system supports block-level de-duplication. This originated on Solaris, but I think FreeBSD supports it, and if you Google around a bit I'm sure you can find a project to port ZFS to Linux. Might also be worth checking the Wikipedia page on file systems, see if any others support block-level deduplication.

    --
    David Hicks
    Nice, Opensolaris here i come.

  4. #4
    apaton's Avatar
    Join Date
    Jun 2009
    Location
    Kings Norton
    Posts
    283
    Thank Post
    54
    Thanked 106 Times in 87 Posts
    Rep Power
    37
    Quote Originally Posted by j17sparky View Post
    Nice, Opensolaris here i come.
    Hold your horses fellas!

    ZFS Dedupe is available only in the latest development source code only.
    I'm running OpenSolaris build 127 which is the latest binary update and that doesn't have it available.

    I'm sure it will be out in a week or two, but this is fresh new code so I wouldn't go production just yet!!!


    Andy

  5. #5
    apaton's Avatar
    Join Date
    Jun 2009
    Location
    Kings Norton
    Posts
    283
    Thank Post
    54
    Thanked 106 Times in 87 Posts
    Rep Power
    37
    ZFS Dedupe delayed due to bug in Fletcher-4 and hash collisions.

    We now have to wait to build 129

    With luck Santa will bring dedupe for Christmas.!

    Andy

  6. #6
    apaton's Avatar
    Join Date
    Jun 2009
    Location
    Kings Norton
    Posts
    283
    Thank Post
    54
    Thanked 106 Times in 87 Posts
    Rep Power
    37
    Well we didn't have to wait for OpenSolaris 129, they built 128a.

    The following example is coping Xvnc 25 times in to a dedup and non-dedup enabled ZFS datasets.
    Code:
    apaton@vb-osol:~$ uname -a
    SunOS vb-osol 5.11 snv_128a i86pc i386 i86pc Solaris
    
    apaton@vb-osol:~$ pfexec mkfile -n 3g /disk1
    apaton@vb-osol:~$ pfexec zpool create tank /disk1
    apaton@vb-osol:~$ pfexec zfs create tank/track
    apaton@vb-osol:~$ pfexec zfs set dedup=on tank/track
    apaton@vb-osol:~$ pfexec zfs create tank/girl
    apaton@vb-osol:~$ pfexec chmod 777 /tank/track /tank/girl
     
    apaton@vb-osol:~$ cd /tank/track
    apaton@vb-osol:/tank/track$ for i in 1 2 3 4 5  ; do mkdir $i; for p in 1 2 3 4 5  ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done
    apaton@vb-osol:/tank/track$ cd /tank/girl
     apaton@vb-osol:/tank/girl$ for i in 1 2 3 4 5  ; do mkdir $i; for p in 1 2 3 4 5  ; do cp /usr/bin/Xvnc $i/Xvnc.$p; done; done
    
    apaton@vb-osol:/tank/girl$ df -h | egrep "(^File|^tank)"
    Filesystem            Size  Used Avail Use% Mounted on
    tank                  2.9G   24K  2.9G   1% /tank
    tank/girl             3.0G   85M  2.9G   3% /tank/girl
    tank/track            3.0G   85M  2.9G   3% /tank/track
    apaton@vb-osol:/tank/girl$ zfs list -r tank
    NAME         USED  AVAIL  REFER  MOUNTPOINT
    tank         169M  2.85G    24K  /tank
    tank/girl   84.5M  2.85G  84.5M  /tank/girl
    tank/track  84.5M  2.85G  84.5M  /tank/track
    apaton@vb-osol:/tank/girl$ zpool list tank
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    tank   2.98G  88.2M  2.90G     2%  25.00x  ONLINE  -
    
    I copied Xvnc 155 times then noticed not only is dedup working perfectly but my filesystem /tank/track grew!! Are we using 16% of the 3G ZFS pool?

    This might be misleading in you use df in your filesystems monitoring scripts.

    Code:
    apaton@vb-osol:/tank/track$ find . -name Xvnc\* | wc -l
    155
    
    apaton@vb-osol:/tank/track$ zpool list tank
    NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
    tank   2.98G  88.7M  2.90G     2%  155.00x  ONLINE  -
    
    apaton@vb-osol:/tank/track$ df -h | egrep "(^File|^tank)"
    Filesystem            Size  Used Avail Use% Mounted on
    tank                  2.9G   24K  2.9G   1% /tank
    tank/girl             3.0G   85M  2.9G   3% /tank/girl
    tank/track            3.4G  524M  2.9G  16% /tank/track
    
    apaton@vb-osol:/tank/track$ zfs list -r tank
    NAME         USED  AVAIL  REFER  MOUNTPOINT
    tank         608M  2.84G    24K  /tank
    tank/girl   84.5M  2.84G  84.5M  /tank/girl
    tank/track   524M  2.84G   524M  /tank/track
    Last edited by apaton; 7th December 2009 at 08:38 AM. Reason: typo

  7. #7


    Join Date
    Oct 2006
    Posts
    3,414
    Thank Post
    184
    Thanked 356 Times in 285 Posts
    Rep Power
    149
    Quote Originally Posted by apaton View Post
    Well we didn't have to wait for OpenSolaris 129, they built 128a.


    you know what im doing tomorrow then



SHARE:
+ Post New Thread

Similar Threads

  1. Sun Storage 7110 Backup Options
    By RobFuller in forum Hardware
    Replies: 50
    Last Post: 23rd October 2009, 09:44 AM
  2. Xenserver 5.5 with Sun Storage 7110 backup?
    By cookie_monster in forum Thin Client and Virtual Machines
    Replies: 2
    Last Post: 12th August 2009, 06:30 AM
  3. Replies: 1
    Last Post: 24th July 2009, 01:37 PM
  4. Active Directory Duplication
    By azrael78 in forum Windows Server 2000/2003
    Replies: 0
    Last Post: 3rd April 2009, 04:57 PM
  5. Backup program for Apple Servers to remote storage
    By ranj in forum How do you do....it?
    Replies: 0
    Last Post: 19th May 2008, 10:55 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •