Archiving Data - what do you do?

[Sohailahmad]

Hello all

 

I have a couple of schools with data storage issues. They seem to be filling the free space at a crazy rate. Recently I've tried to clear as much non-user data up as much as I can. At one point I was able to free 120GB....4 weeks later it was gone.

 

With the introduction of mobile devices free server space has become a rare commodity. Pictures and video have become the bane of my life. To be honest, pictures not as much since I now batch process them, anything over a certain size I now shrunk to something more reasonable. I run this now and then, warning staff if they want to keep anything it's original size (why they would want to I don't know) then move it to a different location. I then batch process the user data area.

 

The other issue come when staff transfer pictures from the device to the server, not clearing out blurry pictures or similar images, they don't stop there...they then COPY a whole load from one area to another duplicating huge amounts. Again I have found software that can identify duplicates.

 

Do anyone of you have any quick solutions for the videos?

 

General data is also an issue, old files just end up sitting there not being used, what do you all do to archive data that the school don't want deleted but to be archived and referred to later?

 

Any other tips on housekeeping on the server to keep as much free space available as possible?

 

thanks in advance

[3s-gtech]

Set up de-duplication on the file server if possible (Server 2012 and on for Windows servers). This will save a huge amount of space and hassle. You can batch convert videos too using something like Handbrake. Also, quotas.

[Roopert]

HDGraph is a really good tool for finding space usage, it is likely to only be certain staff if it is video's or photo's.

 

This will give you a good idea of who is using the space so you can approach them directly or via SLT.

 

It is free and can run off a memory stick and search any drive you set with a handy graphical output.

 

I won't link the software *your chosen search engine* is on hand!

 

Another culprit in the past has been apple profiles and cache locations - might be worth checking.

Edited June 8, 2016 by Roopert

[Sohailahmad]

Set up de-duplication on the file server if possible (Server 2012 and on for Windows servers). This will save a huge amount of space and hassle. You can batch convert videos too using something like Handbrake. Also, quotas.

 

Unfortunately they're 2008 R2. I have seen deduplication working while training, looks like it works well. I'll have a look at handbrake and see what options it has, thank you.

 

HDGraph is a really good tool for finding space usage, it is likely to only be certain staff if it is video's or photo's.

 

This will give you a good idea of who is using the space so you can approach them directly or via SLT.

 

It is free and can run off a memory stick and search any drive you set with a handy graphical output.

 

I won't link the software *your chosen search engine* is on hand!

 

Another culprit in the past has been apple profiles and cache locations - might be worth checking.

 

I use JDisk to gather information about where all the data lies, i'll try out HDGraph and see it it's better.

 

I normally do name and shame, sometimes it helps but in the long term it just ends up being someone else and also in this case, since the school have 90 iPads, 3 sets and the rest staff they all seem to be filling the server.

 

Yes Apple are terrible with roaming app data, I always delete any iOS updates I find. But again, I've got that covered as the iPads are deployed via an MDM and don't need iTunes to update, so rarely when I do find one, it's the staff members personal device.

 

Any ideas with archiving old data? I was thinking on purchasing external HDs and moving data according to year then locking them away in the safe, good? bad?

[3s-gtech]

For archiving, I use a large Synology NAS (expanded up to 36TB now but that's with an add-on). This has enough space to hold all of our archive data. The original unit was around £1000 fully populated with disks.

[Sohailahmad]

For archiving, I use a large Synology NAS (expanded up to 36TB now but that's with an add-on). This has enough space to hold all of our archive data. The original unit was around £1000 fully populated with disks.

 

What RAID have you used? what about the risk of data loss in case the unit dies?

[kathabell]

We run Treesize Professional regularly to ascertain who-is-using-what and tackle the individuals. We offer to bulk compress photographs and/or videos and archive data to disk (CD, DVD or USB, whatever they prefer) if they wish. Inevitably with the use of iPads and increase in media/video work our requirements are climbing, but at least we are attempting to keep it to realistic proportions. In the past we have used disk quotas and if necessary would consider returning to them - but we'd prefer not to if possible.

[mavhc]

Buy some 8TB HDDs

 

Create 1 storage area for photos, with a folder for each year, 2014, 2015, 2016, inside one for each month, 01 02 03, and inside one for each day/class/event, 31 School Fair

 

For videos from mobile devices you can approx halve the storage space with https://github.com/mavhc/shrink-video-recursive-python

 

Train users to delete photos from devices to avoid duplicates. Set quotas if they don't. Wipe devices regularly.

 

Use Windirstat

[3s-gtech]

What RAID have you used? what about the risk of data loss in case the unit dies?

 

RAID handled by Synology's own default SHR setup on the device (with 1 disk tolerance). Data loss? Meh. I've archived it for a reason - it's always old data on here that hasn't been touched for a good while. It can be separately backed up if desired, but there comes a point where you have to draw a line for cost reasons.

[Sohailahmad]

Excellent, lots of suggestions, thank you all. If there is anything else, keep them coming!