CyberNerd Posted May 17, 2012 Posted May 17, 2012 Though I'd share this, as it is a common problem. It saved us 40GB space on the first run. We took the decision to shrink and compress all jpegs over 14 months and >1MB old on our shared drives. This is how we do it. find ./ -type f -size +1M -mtime +426 -iname "*.jpg" -exec convert -resize 60X60% -quality 60 {} {} \; for windows users just install Cygwin 1
hit Posted May 17, 2012 Posted May 17, 2012 (edited) Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller? The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased! Edited May 17, 2012 by hit 1
CyberNerd Posted May 17, 2012 Author Posted May 17, 2012 Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller? The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased! It compresses and resizes (the quality argument) The idea is to keep image sizes down to <1MB. Only images that have not been modified for 14months are affected, so it is a gradual process. Our art department are in full agreement because they don't need to be able to manipulate or print poster sized photographs over that age.
jamesfed Posted May 17, 2012 Posted May 17, 2012 I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups. Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering. Long live 2TB SATA!
hit Posted May 17, 2012 Posted May 17, 2012 Our art department are in full agreement because they don't need to be able to manipulate or print poster sized photographs over that age. your a lucky person, our arts depts are nuts and want everything stored, but I guess my schools are all special needs and photo/video/audio evidence is a must. I built a 12Tb freenas box in the end just to serve media content.
CyberNerd Posted May 17, 2012 Author Posted May 17, 2012 I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups. Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering. Long live 2TB SATA! it did only take a minute to write. I thought I'd share to help out others. I did some dedupe experiments on a similar dataset (here: http://www.edugeek.net/forums/nix/94400-deduplication-mapped-drives.html ) The dudupe ratio was 1.16% if I included the staff homedirectories. Extrapolating this current experiment I achieve nearer 1.10% just with jpeg compression/resizing. Video is the real problem. I'm half tempted to put some on our Google apps video channel. Long live 2TB SATA! 1GB IBM drives are still costing me nearly £200 ea at the moment. Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering. your a lucky person, our arts depts are nuts and want everything stored, but I guess my schools are all special needs and photo/video/audio evidence is a must. I built a 12Tb freenas box in the end just to serve media content. My 14month plan is very similar to your LEA strategy. I just can't see someone needing a full resolution picture years down the road. perhaps they will, at which point I'll refer them to the SLA lol. 1
sidewinder Posted May 17, 2012 Posted May 17, 2012 1GB IBM drives are still costing me nearly £200 ea at the moment. Are you stuck in 1996?
CyberNerd Posted May 17, 2012 Author Posted May 17, 2012 Are you stuck in 1996? Must be. IBM do produce fantastic SAN, notwithstanding the drive prices: https://www.google.co.uk/webhp?sourceid=chrome-instant&ix=sea&ie=UTF-8&ion=1#hl=en&q=43W7630&ix=sea&ion=1&um=1&ie=UTF-8&tbo=u&tbm=shop&source=og&sa=N&tab=wf&ei=4We1T4fdNoma8gOyvpnMDw&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=fba838c2d080fe1e&biw=1280&bih=684
Arthur Posted May 17, 2012 Posted May 17, 2012 Converting BMPs to PNGs can save a huge amount of space too. for /R %%i in (*.bmp) do bmp2png -9 -E -T "%%i" 1
CyberNerd Posted May 17, 2012 Author Posted May 17, 2012 Converting BMPs to PNGs can save a huge amount of space too. for /R %%i in (*.bmp) do bmp2png -9 -E -T "%%i" yes thanks. bitmap is a terrible format. I checked them already and we have very few on our filesystems. 1
gshaw Posted May 18, 2012 Posted May 18, 2012 I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups. Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering. Long live 2TB SATA! Storage space isn't cheap at the moment... certainly not at the levels it was say 12 months ago
zag Posted May 18, 2012 Posted May 18, 2012 We have done this for a number of years, saves about 10gb a term for us. No one has ever noticed in the 6 years I've been doing it. On windows 7 you can just search for all the jpgs on a shared drive like this in the file explorer search box size:large *.jpg Then use Image Resizer for Windows to just right click and resize the images. In my experience it saves around 90% of the size by resizing to Large. 1
zag Posted May 18, 2012 Posted May 18, 2012 I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups. Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering. Been archiving files older than 5 years for a while as well. This robocopy command will do it robocopy.exe "E:\Data\TeacherDataNew" "E:\Data\_Archive\TeacherDataNew" *.* /S /COPY:DATSO /MOVE /ZB /NP /XA:RHS /XF *.ini *.ico /MINAGE:1800 /R:2 /W:10 Again, no one has ever really noticed, and if they do I just move their specific files back. This keeps our entire data collection for a 1000 user school below 300gb
ste1988 Posted May 18, 2012 Posted May 18, 2012 We have done this for a number of years, saves about 10gb a term for us. No one has ever noticed in the 6 years I've been doing it. On windows 7 you can just search for all the jpgs on a shared drive like this in the file explorer search box size:large *.jpg Then use Image Resizer for Windows to just right click and resize the images. In my experience it saves around 90% of the size by resizing to Large. Just found about 10,000 jpgs over 1mb using that, thanks very much, all just resized using image resizer. Have been thinking of getting a nas just for images and video, we just have 2tb external drives atm for archieving pervious years media.
jinnantonnixx Posted May 18, 2012 Posted May 18, 2012 (edited) Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller? The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased! A way around this would be to add '-exec rename ...' to add a flag to a resized file. e.g. lolcat_rs.jpg. Adding _rs to the file flags it as resized. Then add a condition to the find to ignore the _rs flag. e.g.( .... ! -iname "*_rs.jpg") . Just a thought.... Edited May 18, 2012 by jinnantonnixx
CyberNerd Posted May 18, 2012 Author Posted May 18, 2012 A way around this would be to add '-exec rename ...' to add a flag to a resized file. e.g. lolcat_rs.jpg. Adding _rs to the file flags it as resized. Then add a condition to the find to ignore the _rs flag. e.g.( .... ! -iname "*_rs.jpg") . Just a thought.... like this: find ./ -type f -size +1M -mtime +426 \( -iname "*.jpg" ! -iname "rs_*" \) -exec convert -resize 60X60% -quality 60 {} rs_{} \;
Pete10141748 Posted May 18, 2012 Posted May 18, 2012 (edited) We have done this for a number of years, saves about 10gb a term for us. No one has ever noticed in the 6 years I've been doing it. On windows 7 you can just search for all the jpgs on a shared drive like this in the file explorer search box size:large *.jpg Then use Image Resizer for Windows to just right click and resize the images. In my experience it saves around 90% of the size by resizing to Large. Holy heck, just ran this on the Shared area here and found a little over 35,000 jpg's! Going to shrink them now, that's ridiculuous! Edited May 18, 2012 by Pete10141748
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now