+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 17
*nix Thread, How can I write a v. simple shell script to back selected files by date in Technical; I'm using Ubuntu 9.04 server (yes, it should be 8.04 LTS) and Moodle 1.9 I'd like to make a 'small' ...
  1. #1

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0

    How can I write a v. simple shell script to back selected files by date

    I'm using Ubuntu 9.04 server (yes, it should be 8.04 LTS) and Moodle 1.9

    I'd like to make a 'small' backup for keeping offsite so I thought I'd just back up the course backup .zip files.

    I've figured out that I can use the command

    sudo cp /path_to/moodledata/[[:digit:]]*/*.zip /backupdirectory

    to copy out just backups from the moodledata directory.

    Now my problem is that I have set to keep 5 backups of each course. I changed this to 1 as a quick workaround but only those courses that get backed up delete the extra backups so I now have somewhat less than 5x the amount of data I want.

    I feel sure there must be a way to write a shell script that would look into each backup directory and select only the newest *.zip file for copying and that then all my troubles will be at an end.

    Can anyone help me?
    Thanks
    Carol

  2. #2

    LosOjos's Avatar
    Join Date
    Dec 2009
    Location
    West Midlands
    Posts
    5,182
    Thank Post
    1,285
    Thanked 1,029 Times in 729 Posts
    Rep Power
    658
    Quote Originally Posted by CarolBooth View Post
    Now my problem is that I have set to keep 5 backups of each course. I changed this to 1 as a quick workaround but only those courses that get backed up delete the extra backups so I now have somewhat less than 5x the amount of data I want.
    Just to clarify, do you want to backup the previous 5 versions of a file, or the previous 5 days regardless of updates?

  3. #3


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    338
    Code:
    find  /path/to/zip/files -type f -name "*.zip" -mtime -5 -exec cp {} /path/to/zip/backups/{} \;
    where mtime = files modified within 5 days.

    also consider using rsync to copy as it will only copy files that have changed

    rsync

  4. #4

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    sorry Josh, that part is more about Moodle than about the script. I only want to keep one (newest) backup from each backup directory but most directories also contain older files.

  5. #5


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    338
    you could also use the 'find' command to delete the older copies - substitute cp with rm as above

  6. #6

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Thanks Cybernerd. I'm afraid I didn't explain very well. Moodle backs up courses into a separate numbered directory for each course. It only does this when they have changed so the latest backup of one course will be dated 60 days ago and another will be dated today. This is why I need to compare with other backups in the same directory.

    Absolutely agree about rsync - I do use that for other backups but I'm taking baby steps here!
    Carol

  7. #7


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    338
    The find command will recursively search directories and look for the newest files.. does that help?

  8. #8

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Quote Originally Posted by CyberNerd View Post
    The find command will recursively search directories and look for the newest files.. does that help?
    Ah, light dawns
    I thought the 5 days was the age of the file. So I'd set that to 60 days if I thought that was the oldest backup I might want to keep (or could I leave it out)?
    Thanks for your very quick response.
    Carol

  9. #9

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Quote Originally Posted by CyberNerd View Post
    The find command will recursively search directories and look for the newest files.. does that help?
    Quote Originally Posted by CarolBooth View Post
    Ah, light dawns
    I thought the 5 days was the age of the file. So I'd set that to 60 days if I thought that was the oldest backup I might want to keep (or could I leave it out)?
    Thanks for your very quick response.
    Carol
    No, lights are out again
    find /path/to/zip/files -type f -name "*.zip" -mtime -5
    gets me all .zip files in the directory that have been modified in the last 5 days.
    How can I pick from those selected files only the most recent one, without knowing the date it was modified?
    Carol

    just found this example to delete all but latest 5 files
    at http://stackoverflow.com/questions/2...-in-bash/25792

    (ls -t|head -n 5;ls)|sort|uniq -u|xargs rm

    ls -t|head -n 5 will list only the newest 5 files
    so the part in brackets lists the newest file plus all files,
    then |sort|unique -u removes the repeated filename (which is the one I want) then |xargs rm sends the other files to the rm command.

    If I replace number 5 with 1 and unique -u with unique --repeated then I should select only the file I want in a directory.

    Now, how to script or write a command to do this for every directory that matches /path/moodledata/[[:digit:*]]/backupdata/
    Last edited by CarolBooth; 10th February 2010 at 02:20 PM. Reason: update

  10. #10

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    I have found a command that will copy the file I want:

    Code:
    ls *.zip -t|head -n 1 |xargs -i cp ./{} /moodle-courses-backup/
    This lists the files in timestamp order but only the first (1) file then xargs copies the file it's been given.
    Now I need to iterate throught each directory matching /path/moodledata/NN/backupdata (where NN is any number of digits) and do the same thing. I'm not sure how to do this using a variable - can anyone advise?

  11. #11

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Below is script that I think will do what I want (the commands work separately) but I'm getting errors:

    Code:
    ./backup-courses.sh: line 8: unexpected EOF while looking for matching `"'
    ./backup-courses.sh: line 12: syntax error: unexpected end of file
    Can you spot my mistake?
    Code:
    #!/bin/bash
    
    LIST="$(ls -d /usr/share/moodledata/[[:digit:]]*/"
    #make a list of all the course data directories
    for i in "$LIST"; do
    	ls "$i"/backupdata/*.zip -t|head -n 1 |xargs -i cp {} /moodle-courses-backup/
    done
    #when this works, replace cp with rsync
    Last edited by CarolBooth; 10th February 2010 at 04:12 PM. Reason: correcting typo

  12. #12


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    338
    I'm still fixated on getting it on one line. I'm sure find can do it!

    try this:
    Code:
    find . -type f -printf "%TY-%Tm-%Td %TT %p\n"  | sort | tail -n 1 | awk '{ print $3 }' | xargs /bin/cp -v   --target-directory=/backups
    "tail -n 1" should print the last line, hence the most recent "tail -n 5" is the 5 most recent
    Last edited by CyberNerd; 10th February 2010 at 08:38 PM. Reason: explanation

  13. Thanks to CyberNerd from:

    CarolBooth (11th February 2010)

  14. #13

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Wow. I'll get my dictionary out and let you know if it works once I've understood it.
    I'm afraid I've got meetings today and then off work until after half term so please don't be offended if I don't get back to you for a while. Thanks for your help CyberNerd.

  15. #14

    Join Date
    Aug 2008
    Location
    Durham
    Posts
    40
    Thank Post
    10
    Thanked 0 Times in 0 Posts
    Rep Power
    0
    Cybernerd, I'm sorry but I can't get your lovely single line to do what I want. I expect I haven't understood what you've given me but I'm now determined to solve this! I wrote a script that really SHOULD work - the bits work on their own at the command line. I have directories names /usr/moodledata/1/backupdata/, /usr/moodledata/2/backupdata all the way up to 300 and I want to get out only the most recent file from each backupdata directory.
    Maybe you can see my simple mistake?

    Code:
    #!/bin/bash
    echo "I'm a script"
    
    #make a list of all the course data directories
    LIST=$(ls -d /usr/moodledata/* | grep '/usr/moodledata/[0-9]')
    
    for i in "$LIST"; do
    #       Debug:  write to file a list of the directories I want to look in:
    #       echo $i > /home/administrator/list
    
          #list all the .zip files in the backup directory and copy the newest
          ls "$i"/backupdata/*.zip -t|head -n 1 |xargs -i cp {} /backup_moodle_courses/
    done
    The output I got was:

    Code:
    I'm a script
    ls: cannot access /usr/moodledata/1
    /usr/moodledata/100
    /usr/moodledata/101
    ... and so on until
    Code:
    /usr/moodledata/300
    Last edited by CarolBooth; 15th February 2010 at 09:16 PM.

  16. #15


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    338
    ok, I see now that you need the latest file from each directory.
    Keeping with my obsessive theme of one-liners.....

    Code:
    for file in $(/bin/ls -d /usr/moodledata/*/backupdata/); do find $file -type f -iname "*.zip" -printf "%TY-%Tm-%Td %TT %p\n"  | sort | tail -n 1 | awk '{ print $3 }' | xargs /bin/cp -v   --target-directory=/backup_moodle_courses/ ; done
    (feel free to put it on more than one line)
    Last edited by CyberNerd; 16th February 2010 at 10:33 AM.

  17. Thanks to CyberNerd from:

    CarolBooth (24th February 2010)

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Simple call centre type script
    By Phillip1983 in forum Coding
    Replies: 0
    Last Post: 17th January 2010, 08:05 PM
  2. Simple PHP mail save script
    By phpelite in forum Scripts
    Replies: 2
    Last Post: 1st December 2009, 12:56 AM
  3. Replies: 1
    Last Post: 9th September 2009, 07:22 PM
  4. simple bash for loop script
    By RabbieBurns in forum Scripts
    Replies: 24
    Last Post: 5th August 2009, 02:07 PM
  5. Replies: 4
    Last Post: 19th March 2009, 02:35 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •