How can I write a v. simple shell script to back selected files by date

[CarolBooth]

I'm using Ubuntu 9.04 server (yes, it should be 8.04 LTS) and Moodle 1.9

 

I'd like to make a 'small' backup for keeping offsite so I thought I'd just back up the course backup .zip files.

 

I've figured out that I can use the command

 

sudo cp /path_to/moodledata/[[:digit:]]*/*.zip /backupdirectory

 

to copy out just backups from the moodledata directory.

 

Now my problem is that I have set to keep 5 backups of each course. I changed this to 1 as a quick workaround but only those courses that get backed up delete the extra backups so I now have somewhat less than 5x the amount of data I want.

 

I feel sure there must be a way to write a shell script that would look into each backup directory and select only the newest *.zip file for copying and that then all my troubles will be at an end.

 

Can anyone help me?

Thanks

Carol

[LosOjos]

Now my problem is that I have set to keep 5 backups of each course. I changed this to 1 as a quick workaround but only those courses that get backed up delete the extra backups so I now have somewhat less than 5x the amount of data I want.

 

Just to clarify, do you want to backup the previous 5 versions of a file, or the previous 5 days regardless of updates?

[CyberNerd]

find  /path/to/zip/files -type f -name "*.zip" -mtime -5 -exec cp {} /path/to/zip/backups/{} \;

 

where mtime = files modified within 5 days.

 

also consider using rsync to copy as it will only copy files that have changed

 

rsync

[CarolBooth]

sorry Josh, that part is more about Moodle than about the script. I only want to keep one (newest) backup from each backup directory but most directories also contain older files.

[CyberNerd]

you could also use the 'find' command to delete the older copies - substitute cp with rm as above

[CarolBooth]

Thanks Cybernerd. I'm afraid I didn't explain very well. Moodle backs up courses into a separate numbered directory for each course. It only does this when they have changed so the latest backup of one course will be dated 60 days ago and another will be dated today. This is why I need to compare with other backups in the same directory.

 

Absolutely agree about rsync - I do use that for other backups but I'm taking baby steps here!

Carol

[CyberNerd]

The find command will recursively search directories and look for the newest files.. does that help?

[CarolBooth]

The find command will recursively search directories and look for the newest files.. does that help?

 

Ah, light dawns

I thought the 5 days was the age of the file. So I'd set that to 60 days if I thought that was the oldest backup I might want to keep (or could I leave it out)?

Thanks for your very quick response.

Carol

[CarolBooth]

The find command will recursively search directories and look for the newest files.. does that help?

 

Ah, light dawns

I thought the 5 days was the age of the file. So I'd set that to 60 days if I thought that was the oldest backup I might want to keep (or could I leave it out)?

Thanks for your very quick response.

Carol

 

No, lights are out again

find /path/to/zip/files -type f -name "*.zip" -mtime -5

gets me all .zip files in the directory that have been modified in the last 5 days.

How can I pick from those selected files only the most recent one, without knowing the date it was modified?

Carol

 

just found this example to delete all but latest 5 files

at http://stackoverflow.com/questions/25785/delete-all-but-the-most-recent-x-files-in-bash/25792

 

(ls -t|head -n 5;ls)|sort|uniq -u|xargs rm

 

ls -t|head -n 5 will list only the newest 5 files

so the part in brackets lists the newest file plus all files,

then |sort|unique -u removes the repeated filename (which is the one I want) then |xargs rm sends the other files to the rm command.

 

If I replace number 5 with 1 and unique -u with unique --repeated then I should select only the file I want in a directory.

 

Now, how to script or write a command to do this for every directory that matches /path/moodledata/[[:digit:*]]/backupdata/

Edited February 10, 2010 by CarolBooth
update

[CarolBooth]

I have found a command that will copy the file I want:

 

ls *.zip -t|head -n 1 |xargs -i cp ./{} /moodle-courses-backup/

 

This lists the files in timestamp order but only the first (1) file then xargs copies the file it's been given.

Now I need to iterate throught each directory matching /path/moodledata/NN/backupdata (where NN is any number of digits) and do the same thing. I'm not sure how to do this using a variable - can anyone advise?

[CarolBooth]

Below is script that I think will do what I want (the commands work separately) but I'm getting errors:

 

./backup-courses.sh: line 8: unexpected EOF while looking for matching `"'
./backup-courses.sh: line 12: syntax error: unexpected end of file

Can you spot my mistake?

#!/bin/bash

LIST="$(ls -d /usr/share/moodledata/[[:digit:]]*/"
#make a list of all the course data directories
for i in "$LIST"; do
ls "$i"/backupdata/*.zip -t|head -n 1 |xargs -i cp {} /moodle-courses-backup/
done
#when this works, replace cp with rsync

Edited February 10, 2010 by CarolBooth
correcting typo

[CyberNerd]

I'm still fixated on getting it on one line. I'm sure find can do it!

 

try this:

find . -type f -printf "%TY-%Tm-%Td %TT %p\n"  | sort | tail -n 1 | awk '{ print $3 }' | xargs /bin/cp -v   --target-directory=/backups

 

"tail -n 1" should print the last line, hence the most recent "tail -n 5" is the 5 most recent

Edited February 10, 2010 by CyberNerd
explanation

[CarolBooth]

Wow. I'll get my dictionary out and let you know if it works once I've understood it.

I'm afraid I've got meetings today and then off work until after half term so please don't be offended if I don't get back to you for a while. Thanks for your help CyberNerd.

[CarolBooth]

Cybernerd, I'm sorry but I can't get your lovely single line to do what I want. I expect I haven't understood what you've given me but I'm now determined to solve this! I wrote a script that really SHOULD work - the bits work on their own at the command line. I have directories names /usr/moodledata/1/backupdata/, /usr/moodledata/2/backupdata all the way up to 300 and I want to get out only the most recent file from each backupdata directory.

Maybe you can see my simple mistake?

 

#!/bin/bash
echo "I'm a script"

#make a list of all the course data directories
LIST=$(ls -d /usr/moodledata/* | grep '/usr/moodledata/[0-9]')

for i in "$LIST"; do
#       Debug:  write to file a list of the directories I want to look in:
#       echo $i > /home/administrator/list

     #list all the .zip files in the backup directory and copy the newest
     ls "$i"/backupdata/*.zip -t|head -n 1 |xargs -i cp {} /backup_moodle_courses/
done

 

The output I got was:

 

I'm a script
ls: cannot access /usr/moodledata/1
/usr/moodledata/100
/usr/moodledata/101

... and so on until

/usr/moodledata/300

Edited February 15, 2010 by CarolBooth

[CyberNerd]

ok, I see now that you need the latest file from each directory.

Keeping with my obsessive theme of one-liners.....

 

for file in $(/bin/ls -d /usr/moodledata/*/backupdata/); do find $file -type f -iname "*.zip" -printf "%TY-%Tm-%Td %TT %p\n"  | sort | tail -n 1 | awk '{ print $3 }' | xargs /bin/cp -v   --target-directory=/backup_moodle_courses/ ; done

 

(feel free to put it on more than one line)

Edited February 16, 2010 by CyberNerd

[CarolBooth]

That works great Cybernerd. Thanks very much. Would you mind if I post this (crediting you) or link to this on the Moodle.org site for other Moodle admins?

 

I notice that each time no .zip file exists, it outputs

/bin/cp: missing file operand
Try `/bin/cp --help' for more information.

I guess that doesn't matter?

 

Thanks for all your help

Carol

[CyberNerd]

of course, link back here. I hope it helps others.

 

I don't think the missing .zip is an issue