+ Post New Thread
Page 1 of 2 12 LastLast
Results 1 to 15 of 17
Scripts Thread, Saved 40GB space on shared drives in Coding and Web Development; Though I'd share this, as it is a common problem. It saved us 40GB space on the first run. We ...
  1. #1


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339

    Saved 40GB space on shared drives

    Though I'd share this, as it is a common problem. It saved us 40GB space on the first run.

    We took the decision to shrink and compress all jpegs over 14 months and >1MB old on our shared drives.
    This is how we do it.

    Code:
    find ./ -type f -size +1M -mtime +426 -iname "*.jpg" -exec convert  -resize 60X60% -quality 60 {} {} \;

    for windows users just install Cygwin

  2. Thanks to CyberNerd from:

    speckytecky (18th May 2012)

  3. #2
    hit
    hit is offline
    hit's Avatar
    Join Date
    Mar 2008
    Location
    London
    Posts
    325
    Thank Post
    48
    Thanked 50 Times in 48 Posts
    Rep Power
    51
    Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller?

    The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased!
    Last edited by hit; 17th May 2012 at 09:18 PM.

  4. Thanks to hit from:

    speckytecky (18th May 2012)

  5. #3


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by hit View Post
    Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller?

    The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased!
    It compresses and resizes (the quality argument)
    The idea is to keep image sizes down to <1MB. Only images that have not been modified for 14months are affected, so it is a gradual process. Our art department are in full agreement because they don't need to be able to manipulate or print poster sized photographs over that age.

  6. #4
    jamesfed's Avatar
    Join Date
    Sep 2009
    Location
    Reading
    Posts
    2,207
    Thank Post
    137
    Thanked 345 Times in 291 Posts
    Rep Power
    87
    I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups.
    Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering.

    Long live 2TB SATA!

  7. #5
    hit
    hit is offline
    hit's Avatar
    Join Date
    Mar 2008
    Location
    London
    Posts
    325
    Thank Post
    48
    Thanked 50 Times in 48 Posts
    Rep Power
    51
    Quote Originally Posted by CyberNerd View Post
    Our art department are in full agreement because they don't need to be able to manipulate or print poster sized photographs over that age.
    your a lucky person, our arts depts are nuts and want everything stored, but I guess my schools are all special needs and photo/video/audio evidence is a must. I built a 12Tb freenas box in the end just to serve media content.

  8. #6


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by jamesfed View Post
    I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups.
    Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering.

    Long live 2TB SATA!
    it did only take a minute to write. I thought I'd share to help out others.

    I did some dedupe experiments on a similar dataset (here: deduplication of mapped drives )
    The dudupe ratio was 1.16% if I included the staff homedirectories. Extrapolating this current experiment I achieve nearer 1.10% just with jpeg compression/resizing.
    Video is the real problem. I'm half tempted to put some on our Google apps video channel.



    Quote Originally Posted by jamesfed View Post
    Long live 2TB SATA!
    1GB IBM drives are still costing me nearly £200 ea at the moment.

    Quote Originally Posted by jamesfed View Post
    Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering.
    Quote Originally Posted by hit View Post
    your a lucky person, our arts depts are nuts and want everything stored, but I guess my schools are all special needs and photo/video/audio evidence is a must. I built a 12Tb freenas box in the end just to serve media content.
    My 14month plan is very similar to your LEA strategy. I just can't see someone needing a full resolution picture years down the road. perhaps they will, at which point I'll refer them to the SLA lol.

  9. Thanks to CyberNerd from:

    speckytecky (18th May 2012)

  10. #7

    Join Date
    Jul 2006
    Location
    London
    Posts
    2,962
    Thank Post
    159
    Thanked 152 Times in 116 Posts
    Rep Power
    49
    Quote Originally Posted by CyberNerd View Post
    1GB IBM drives are still costing me nearly £200 ea at the moment.
    Are you stuck in 1996?

  11. #8


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by sidewinder View Post
    Are you stuck in 1996?
    Must be. IBM do produce fantastic SAN, notwithstanding the drive prices:
    https://www.google.co.uk/webhp?sourc...w=1280&bih=684

  12. #9


    Join Date
    Feb 2007
    Location
    51.403651, -0.515458
    Posts
    9,064
    Thank Post
    232
    Thanked 2,717 Times in 2,005 Posts
    Rep Power
    795
    Converting BMPs to PNGs can save a huge amount of space too.

    Code:
    for /R %%i in (*.bmp) do bmp2png -9 -E -T "%%i"

  13. Thanks to Arthur from:

    speckytecky (18th May 2012)

  14. #10


    Join Date
    Jan 2006
    Posts
    8,202
    Thank Post
    442
    Thanked 1,032 Times in 812 Posts
    Rep Power
    339
    Quote Originally Posted by Arthur View Post
    Converting BMPs to PNGs can save a huge amount of space too.

    Code:
    for /R %%i in (*.bmp) do bmp2png -9 -E -T "%%i"
    yes thanks. bitmap is a terrible format. I checked them already and we have very few on our filesystems.

  15. Thanks to CyberNerd from:

    bobkerr (18th May 2012)

  16. #11
    gshaw's Avatar
    Join Date
    Sep 2007
    Location
    Essex
    Posts
    2,662
    Thank Post
    166
    Thanked 220 Times in 203 Posts
    Rep Power
    67
    Quote Originally Posted by jamesfed View Post
    I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups.
    Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering.

    Long live 2TB SATA!
    Storage space isn't cheap at the moment... certainly not at the levels it was say 12 months ago

  17. #12
    zag
    zag is offline
    zag's Avatar
    Join Date
    Mar 2007
    Posts
    3,829
    Thank Post
    918
    Thanked 422 Times in 355 Posts
    Blog Entries
    12
    Rep Power
    88
    We have done this for a number of years, saves about 10gb a term for us. No one has ever noticed in the 6 years I've been doing it.

    On windows 7 you can just search for all the jpgs on a shared drive like this in the file explorer search box

    Code:
    size:large *.jpg
    Then use Image Resizer for Windows to just right click and resize the images. In my experience it saves around 90% of the size by resizing to Large.

  18. Thanks to zag from:

    ste1988 (18th May 2012)

  19. #13
    zag
    zag is offline
    zag's Avatar
    Join Date
    Mar 2007
    Posts
    3,829
    Thank Post
    918
    Thanked 422 Times in 355 Posts
    Blog Entries
    12
    Rep Power
    88
    Quote Originally Posted by jamesfed View Post
    I've looked at doing something very similar a number of times in the past but it all ends up at - why am I bothering? Storage space is cheap and de dupe backup means not much space wasted on the backups.
    Our LEA has recently moved to a 'everything over 5 years old gets moved to tape and if you want it expect the file to take some time to load' methodology but in looking at similar solutions its all just too expensive to be worth bothering.
    Been archiving files older than 5 years for a while as well. This robocopy command will do it

    Code:
    robocopy.exe "E:\Data\TeacherDataNew" "E:\Data\_Archive\TeacherDataNew" *.* /S /COPY:DATSO /MOVE /ZB /NP /XA:RHS /XF *.ini *.ico /MINAGE:1800 /R:2 /W:10
    Again, no one has ever really noticed, and if they do I just move their specific files back.

    This keeps our entire data collection for a 1000 user school below 300gb

  20. #14
    ste1988's Avatar
    Join Date
    Jan 2012
    Location
    West Midlands
    Posts
    177
    Thank Post
    77
    Thanked 24 Times in 23 Posts
    Rep Power
    13
    Quote Originally Posted by zag View Post
    We have done this for a number of years, saves about 10gb a term for us. No one has ever noticed in the 6 years I've been doing it.

    On windows 7 you can just search for all the jpgs on a shared drive like this in the file explorer search box

    Code:
    size:large *.jpg

    Then use Image Resizer for Windows to just right click and resize the images. In my experience it saves around 90% of the size by resizing to Large.


    Just found about 10,000 jpgs over 1mb using that, thanks very much, all just resized using image resizer.

    Have been thinking of getting a nas just for images and video, we just have 2tb external drives atm for archieving pervious years media.

  21. #15

    jinnantonnixx's Avatar
    Join Date
    Mar 2011
    Location
    In the Calamatorium.
    Posts
    1,991
    Thank Post
    114
    Thanked 507 Times in 342 Posts
    Blog Entries
    2
    Rep Power
    286
    Quote Originally Posted by hit View Post
    Surely that's not compressing, that's resizing? What happens next time you run it, doesn't the image get even smaller?

    The way I see this running is after just 3 runs a hi res image would be only 20% of its original dimensions at which point it may be under the 1Mb and not get resized any more but I'm sure our art department wouldnt be pleased!
    A way around this would be to add '-exec rename ...' to add a flag to a resized file. e.g. lolcat_rs.jpg. Adding _rs to the file flags it as resized. Then add a condition to the find to ignore the _rs flag. e.g.( .... ! -iname "*_rs.jpg") .

    Just a thought....
    Last edited by jinnantonnixx; 18th May 2012 at 10:06 AM.

SHARE:
+ Post New Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Replies: 8
    Last Post: 5th January 2012, 08:15 AM
  2. recommended free space on system drive
    By RabbieBurns in forum Windows 7
    Replies: 7
    Last Post: 9th November 2011, 10:26 AM
  3. Run out of space on Server C drive!
    By phantomphan in forum Wireless Networks
    Replies: 39
    Last Post: 22nd June 2011, 01:03 PM
  4. Office 2007 installed on shared drive?
    By Ben-BSH in forum Office Software
    Replies: 7
    Last Post: 23rd March 2010, 12:22 PM
  5. Out of drive space on domain?
    By Samson in forum Windows
    Replies: 7
    Last Post: 5th June 2007, 10:14 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •