+ Post New Thread
Results 1 to 11 of 11
*nix Thread, Help with LVM snapshot scripts for Samba Shadow Copy support in Technical; I need a hand here, my scripting skills are insufficent to pull this one off. However the end result is ...
  1. #1

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,820
    Thank Post
    110
    Thanked 590 Times in 511 Posts
    Blog Entries
    1
    Rep Power
    226

    Help with LVM snapshot scripts for Samba Shadow Copy support

    I need a hand here, my scripting skills are insufficent to pull this one off. However the end result is likely to benifit others in the future. But first, some background info.

    I have recently been creating a new file server running Ubuntu 8.04 on a HP DL140G3 with an external MSA60 Disk Array. Obviously to service the Windows clients this is running Samba (with all the ACL, quota, etc bells and whistles).

    The underlying file system for the user storage is RAID-5. This is then allocated with LVM2 into various partitions. These are formatted as JFS and mounted some place sensible (/home/samba/blah normally) and then shared via samba as required.

    I wish to use LVM based snapshots to provide 'shadow copies' to Windows clients. To do this I need a suite of scripts to do the following:

    1) create LVM snapshots on a regular basis of some LVM paritions. New snapshots must be mounted in the way sambas shadow copy VFS module expects.
    2) remount LVM snapshots cleanly in the case of a server reboot/crash/etc.
    2) delete LVM snapshots after a predetermined time and/or when disk space is low.

    A couple of links with background info:

    Chapter23.Stackable VFS modules
    Samba Shadow Copy Howto - Waikato Linux Users Group
    Samba and MS Shadow Copy

  2. #2

    Ric_'s Avatar
    Join Date
    Jun 2005
    Location
    London
    Posts
    7,592
    Thank Post
    109
    Thanked 770 Times in 598 Posts
    Rep Power
    183
    IIRC AMANDA (see Amanda Open Source Backup ) pretty much does this for you... have you checked it out before re-inventing the wheel?

  3. #3

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,820
    Thank Post
    110
    Thanked 590 Times in 511 Posts
    Blog Entries
    1
    Rep Power
    226
    I searched the site and only came up with a couple of articles mentioning snapshotting MySQL.

  4. #4

    Ric_'s Avatar
    Join Date
    Jun 2005
    Location
    London
    Posts
    7,592
    Thank Post
    109
    Thanked 770 Times in 598 Posts
    Rep Power
    183
    I have been and had a look... since I last looked at it a couple of years ago, a commercial arm has broken off and the web-interface features that I thought could have been used to solve your problem are only in the paid-for version.

  5. #5

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,684
    Thank Post
    1,268
    Thanked 789 Times in 686 Posts
    Rep Power
    237
    Quote Originally Posted by Geoff View Post
    Create LVM snapshots on a regular basis of some LVM paritions. New snapshots must be mounted in the way sambas shadow copy VFS module expects.
    This is just the example script given in "Chapter 23: Stackable VFS Modules", run every day(?) from a cron job.

    remount LVM snapshots cleanly in the case of a server reboot/crash/etc.
    Just write a script to read all the subdirectories in your mount directory and mount them all, then run that script from your rc.local?

    delete LVM snapshots after a predetermined time and/or when disk space is low.
    Just figure out the modified time of each subdirectory, compare it with the current date, delete it if you need to. For disk space, read all subdirectories, order by date, delete the first, loop until you have enough space.

    How do you mean you're having difficulty writing a script for the above? Does it have to be done as a bash script, can't you simply write it in Perl or Python or something?

    --
    David Hicks

  6. #6
    monkeyx's Avatar
    Join Date
    Nov 2006
    Posts
    365
    Thank Post
    8
    Thanked 52 Times in 41 Posts
    Rep Power
    26
    I have a couple of test systems that use openfiler. This support iscsi, snapshots etc.

    Not sure about support for Mysql though.

  7. #7
    ahuxham's Avatar
    Join Date
    Apr 2008
    Posts
    1,122
    Thank Post
    76
    Thanked 138 Times in 109 Posts
    Rep Power
    31
    Quote Originally Posted by monkeyx View Post
    I have a couple of test systems that use openfiler. This support iscsi, snapshots etc.

    Not sure about support for Mysql though.

    Nice bit of software, thanks for the link!

  8. #8

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,820
    Thank Post
    110
    Thanked 590 Times in 511 Posts
    Blog Entries
    1
    Rep Power
    226
    Quote Originally Posted by dhicks View Post
    This is just the example script given in "Chapter 23: Stackable VFS Modules", run every day(?) from a cron job.
    Well yes, that does work on a single LVM2 volume. However I do not have a single volume. I have 20. Of which 14 require shadow copy support.

    Plus there's no error handling.

    Quote Originally Posted by dhicks View Post
    Just write a script to read all the subdirectories in your mount directory and mount them all, then run that script from your rc.local?
    Er no, I have other volumes that are not on LVM2 (the raid1 arrays on the boot drive(s) for example) and I have LVM2 volumes that do not require shadow copy support. So I can't simply blindly assume every volume in /dev/disk/ or /dev/mapper/ is a LVM2 volume and/or snapshot.

    So I need to be keeping track of which LVM2 volumes I wish to snapshot, and keep track of all their snapshots and the creation time of the snapshots.

    Additionally, if the system has crashed, I need to fsck the volumes prior to mounting. Otherwise the mount fails.

    Quote Originally Posted by dhicks View Post
    Just figure out the modified time of each subdirectory, compare it with the current date, delete it if you need to. For disk space, read all subdirectories, order by date, delete the first, loop until you have enough space.
    My scripting ability fails at this point.

    Quote Originally Posted by dhicks View Post
    How do you mean you're having difficulty writing a script for the above? Does it have to be done as a bash script, can't you simply write it in Perl or Python or something?
    I think this problem is too complex for a bash script. I was thinking Python, but as I said, my skills are limited.

  9. #9

    dhicks's Avatar
    Join Date
    Aug 2005
    Location
    Knightsbridge
    Posts
    5,684
    Thank Post
    1,268
    Thanked 789 Times in 686 Posts
    Rep Power
    237
    Quote Originally Posted by Geoff View Post
    Well yes, that does work on a single LVM2 volume. However I do not have a single volume. I have 20. Of which 14 require shadow copy support.
    You could simply create a list of volumes you want to snapshot, or create a list of volumes you don't want to snapshot and have your script parse the output of "lvs" and skip the ones in that list.

    Er no, I have other volumes that are not on LVM2 (the raid1 arrays on the boot drive(s) for example) and I have LVM2 volumes that do not require shadow copy support. So I can't simply blindly assume every volume in /dev/disk/ or /dev/mapper/ is a LVM2 volume and/or snapshot.
    Sorry, I meant as in how that example script works - it makes a new directory to stick mount points in, so you can assume that all subdirectories in that directory can be mounted.

    I think this problem is too complex for a bash script. I was thinking Python, but as I said, my skills are limited.
    Darned if I can remember how to do loops in bash, anyhow. Wouldn't a Python script, very roughly, be something along the lines of:

    Code:
    #!/usr/bin/python
    # Script to be run from a cron job, once a day, to create snapshots.
    import os
    import time
    volumes = ["volOne", "volTwo", "volEtc"]
    snapName = time.strftime("%Y.%m.%d-%H.%M.%S", time.gmtime())
    for volume in volumes:
    	os.system("xfs_freeze -f /data/shadow_share/")
    	os.system("lvcreate -L10M -s -n "+volume+snapName+" /dev/shadowvol/sh_test")
    	os.system("xfs_freeze -u /data/shadow_share/")
    	os.system("mkdir /data/shadow_share/"+volume+snapName)
    	os.system("mount /dev/shadowvol/"+volume+snapName+" /data/shadow_share/"+volume+snapName+" -onouuid,ro")
    Might want to replace the above calls to os.system with a function that prints the commands out before actually running them, letting you see what the script is going to do before actually doing it.

    --
    David Hicks
    Last edited by dhicks; 31st July 2008 at 10:55 AM. Reason: Woops, need to change bash-style date/time to Python.

  10. #10

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,820
    Thank Post
    110
    Thanked 590 Times in 511 Posts
    Blog Entries
    1
    Rep Power
    226
    Well that's a start. I shall have a tinker and see how far I get. Thanks.

  11. #11
    apaton's Avatar
    Join Date
    Jun 2009
    Location
    Kings Norton
    Posts
    283
    Thank Post
    54
    Thanked 106 Times in 87 Posts
    Rep Power
    36

    Red face

    I'm might come across as a Solaris bigot . But if you want a filesystem that can do snapshots & CIFS/NFS/iSCSI support you must look at OpenSolaris ZFS.

    The guys in the Hardware Forum rave on about Sun Unified Storage S7000 range, which is under the covers is basically OpenSolaris + BUI.

    Snapshot are simple in ZFS and very efficient with space.
    To access a snapshot just simply follow the hidden ".zfs" directory which access all the read only snapshots for the share.

    EXAMPLE ONLY (Assuming zpool and Domain/Workgroup setup has been done)
    Code:
    # Create CIFS share
    
    $ zfs create zpool/UserHomeDirs
    $ zfs set sharesmb=on zpool/UserHomeDirs
    Create snapshot
    Code:
    # Snapshot for Monday
    $ zfs snapshot zpool/UserHomeDirs@UserHomeDirs-Monday
    
    # Snapshot for Tuesday
    $ zfs snapshot zpool/UserHomeDirs@UserHomeDirs-Tuesday
    Create many CIFS shares and snapshot them in one command
    Code:
    $ zfs create zpool/cifs-shares
    $ zfs set sharesmb=on zpool/cifs-shares
    $ zfs create zpool/cifs-shares/share1
    $ zfs create zpool/cifs-shares/share2
    ........
    $ zfs create zpool/cifs-shares/share99
    
    ## Snapshot shares 1 to 99
    $ zfs snapshot -r zpool/cifs-shares@cifs-shares-Monday
    I can't do ZFS functionality justice but it well worth the time researching OpenSolaris & ZFS. A blog that may be helpfull. A Home File server using ZFS

SHARE:
+ Post New Thread

Similar Threads

  1. Shadow Copy Problem
    By mickeyh080 in forum Windows
    Replies: 9
    Last Post: 13th May 2008, 03:39 PM
  2. Ntbackup failing Shadow Copy
    By thegrassisgreener in forum Windows
    Replies: 6
    Last Post: 7th April 2008, 11:48 AM
  3. Replies: 18
    Last Post: 13th March 2008, 01:20 PM
  4. Shadow Copy..?
    By Grommit in forum Windows
    Replies: 4
    Last Post: 13th August 2007, 06:26 PM
  5. Shadow Copy
    By Grommit in forum Windows
    Replies: 5
    Last Post: 29th May 2007, 02:01 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •