Managing space on the network

[saggu]

How do people manage the new academic year folders and space on the network? Here we cannot ask staff to delete any files (even though it is duplicate or quadruplet) because data will be deleted!! Interested to know what set up is used in Primary schools.

[Mr.Ben]

We have moved to SharePoint, but when on prem we had Home folders: quotas for staff (5GB)

Shared Drives: Entire contents more to an Archive folder each year, made read only and staff told to copy anything useful in term 1 (after which it was backed up and removed).

[Guest Guest]

It might be worth enabling deduplication on your file server. We're only a small special school and last I checked it was saving us somewhere in the region of 500Gb

[DrCheese]

We've always quota'd them - When users see a red line on their homedrive they tend to tidy up themselves. We aren't ultra strict tho and will increase the space if people *really* need it.

 

Once we're on onedrive/sharepoint et all I probably won't care

[stevejp]

Please may I make a plea for a little tolerance and understanding of the need (under certain circumstances) for a certain amount of file duplication? In my role as head of English there are many documents (policies, reading records etc) that I need to make available to staff, but where I also need to keep a copy in my own area just in case someone decides to make unauthorised changes. I can't keep them on a USB stick, external hard drive etc for security and GDPR reasons, but I do need to have the facility to be able to return to the original document.

[rich_tech]

Please may I make a plea for a little tolerance and understanding of the need (under certain circumstances) for a certain amount of file duplication? In my role as head of English there are many documents (policies, reading records etc) that I need to make available to staff, but where I also need to keep a copy in my own area just in case someone decides to make unauthorised changes. I can't keep them on a USB stick, external hard drive etc for security and GDPR reasons, but I do need to have the facility to be able to return to the original document.

That is what deduplication combats though there, its excellent technology, we have it where I work(non-edu), multiple members of departments like Marketing/Quality have copies of the same document, that chops its storage down, we saved 120GB on one server and 75gb on another so far, were transitioning our old file server to a dedup capable one at the moment and I reckon that 600GB of storage will see a drop to about a third of that at our estate.

[saggu]

Please may I make a plea for a little tolerance and understanding of the need (under certain circumstances) for a certain amount of file duplication? In my role as head of English there are many documents (policies, reading records etc) that I need to make available to staff, but where I also need to keep a copy in my own area just in case someone decides to make unauthorised changes. I can't keep them on a USB stick, external hard drive etc for security and GDPR reasons, but I do need to have the facility to be able to return to the original document.

 

Thank you for your response.

You can always make the documents write protect. In that way no one can change anything.

What procedure you would recommend for your IT department to manage the network space?

[saggu]

That is what deduplication combats though there, its excellent technology, we have it where I work(non-edu), multiple members of departments like Marketing/Quality have copies of the same document, that chops its storage down, we saved 120GB on one server and 75gb on another so far, were transitioning our old file server to a dedup capable one at the moment and I reckon that 600GB of storage will see a drop to about a third of that at our estate.

@rich_tech, in your experience is there any complications in using deduplication? because the external consultant is not keen in doing this.

One thing I am thinking of is if there are duplicated files in different locations how do we know which one is the original or last edited? and it it the one which is saved and others are deleted?

[rich_tech]

there isnt anything much in turning deduplication on, it by default only touches files over 3 days old on the server if its on the default schedule.

 

If your that worried, get a full backup done first on the storage you are going to switch it onto and try it, you will notice the savings after a few days, it is quite excellent technology.

 

Whats the external consultants purpose if I may ask ? if hes trying to sell you a solution or push you down a particular path i.e. selling storage to you or pushing you onto the cloud platform, hes not likely to go down the cheaper option where you make use of the existing storage in a better way.

 

I would say that as long as you dont have compression enabled on the volumes you want to enable it(thats not recommended), it should be ok.

 

I would say at this point do the reading on it - https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview this is the microsoft section on it and will tell you whats needed including patches before turning it on.

[DrCheese]

That is what deduplication combats though there, its excellent technology,

 

Dedupe in Server 2012 caused me massive headaches when our main file server stealthly dumped errors in it's event log about not being able to run the nightly scrubbing/corruption check (If you read how it works, it essentially corrupts/repairs itself constantly...) which eventually resulted in the majority of our file server becoming corrupted beyond repair. It was not a fun few days restoring from backup trying to find one where the corruption hadn't gone beyond repair. As I had also deduped our backup storage at the time, the restore/rehydrating process took *forever* Especially as it was single threaded in 2012 - (I now use Veeam REFS fast clone tech)

 

Microsoft changed it in 2016 so it would scream at you if it detected it hadn't run the cleanup task...

 

It's improved massively over the years but I'm kinda sworn off the tech now.

Edited July 28, 2019 by DrCheese

[rich_tech]

Dedupe in Server 2012 caused me massive headaches when our main file server stealthly dumped errors in it's event log about not being able to run the nightly scrubbing/corruption check (If you read how it works, it essentially corrupts/repairs itself constantly...) which eventually resulted in the majority of our file server becoming corrupted beyond repair. It was not a fun few days restoring from backup trying to find one where the corruption hadn't gone beyond repair. As I had also deduped our backup storage at the time, the restore/rehydrating process took *forever* Especially as it was single threaded in 2012 - (I now use Veeam REFS fast clone tech)

 

Microsoft changed it in 2016 so it would scream at you if it detected it hadn't run the cleanup task...

 

It's improved massively over the years but I'm kinda sworn off the tech now.

aye, I havent been saying there arent or cannot be issues you definitely need to do some reading first up, Veeam is very worthy of being mentioned as the godsend it can be too, also I would even recommend an Azure files and folders backup these days, it costs peanuts for that and can go back years.

[keyboards]

Home folder x GB

Network shares - trying to push google classroom, drive and so forth as much as possible

[mavhc]

Deduplication on 2012r2 with dynamically expanding virtual drives went horribly, used all the space on the server and broke everything.

 

If people aren't authorised to change a document, they shouldn't have write access to it. Fix that first.

[SVM]

Staff home docs are monitored rather than a forced quota, currently also have onedrive auto signed in they could use, we will be moving to onedrive for all documents shortly

 

Students are limited to 5Gb and can ask for an increase if required

 

Shared areas were moved to Sharepoint a couple of years ago (accessed via browser) soon to be moved to Teams so it can be synced/mounted in explorer with onedrive

[robjduk]

Vote for deduplication here. Saved about 500GB of data here on Server 2012R2 and never had a problem with it. If your real problem is duplicate files then that will fix it overnight.

 

I push the Google Drive and have been moving some shared folders over to Google Team Drive with their consent. I'm making shared folders 5GB tops. I also have a NAS box which I drag stuff into once in a while. You get stuff so obviously not required any more and just drag that over in secret. When you see folders called 2009 archive then you know they won't be missing it.

 

Also slip into conversation with your head about the cost of storage. If they are happy to pay for the space and the backup then to hell with it and just do it.

[jthompson]

For one of our shared drives, I have FSRM run an annual task to expire any files older than 5 years. I then use a freeware utility called RED to remove any empty directories. The stuff that has been expired is simply moved out of the main share, so we can restore anything that people discover they still want.

 

This works well for this particular drive because it's a bit of a general dumping ground, largely full of transient stuff. The main staff shared drive is dealt with differently. I just monitor that for any folders jumping in size significantly from one day to the next (so as to catch anything silly like 1GB PowerPoints, camera dumps, etc.). I monitor that using PA WatchDisk/.

 

User areas are quotad to just a few GB initially, with a higher quota on request. We're pushing staff towards Google Drive with Team Drives and File Stream now, which frees things up considerably.

[mikes]

Are you using roaming profiles? Have you checked the profile sizes as they can build up quickly. Ours weren't configured very well so they were growing to multi-gigabyte sizes; with temp files, OneDrive; outlook attachments etc.

Other than that, I have actually just let space get to 5-10% remaining; once staff see that red bar they tend to go through what they don't need.

I made the mistake before of just keeping expanding the storage drives every time they went red, it looked like we had plenty of free space but we ended up completely filling our SAN and grinding the network to a halt...

 

We also use DeDup here, never had any big problems with it but it can be finicky when you are trying to clean up space. We ran out of space and I needed to quickly clean 50 gigabytes; of course deleting that much data did not free up any space until the next Dedup cycle actually ran (which you can force via commandline)

 

A problem I have regarding staff managing their own space, is that the vast majority of them don't seem to understand the concept of file size. They will see a folder that has 50 word documents in it, totalling about 30MB; they will think this folder is much much bigger than a different folder that only has 2 powerpoints in it (ppt's with images and videos totalling 270MB). So they are good at reducing the number of files they have, but not the amount of data they have.

Edited August 14, 2019 by mikes

[saggu]

Are you using roaming profiles? Have you checked the profile sizes as they can build up quickly. Ours weren't configured very well so they were growing to multi-gigabyte sizes; with temp files, OneDrive; outlook attachments etc.

Other than that, I have actually just let space get to 5-10% remaining; once staff see that red bar they tend to go through what they don't need.

I made the mistake before of just keeping expanding the storage drives every time they went red, it looked like we had plenty of free space but we ended up completely filling our SAN and grinding the network to a halt...

 

We also use DeDup here, never had any big problems with it but it can be finicky when you are trying to clean up space. We ran out of space and I needed to quickly clean 50 gigabytes; of course deleting that much data did not free up any space until the next Dedup cycle actually ran (which you can force via commandline)

A problem I have regarding staff managing their own space, is that the vast majority of them don't seem to understand the concept of file size. They will see a folder that has 50 word documents in it, totalling about 30MB; they will think this folder is much much bigger than a different folder that only has 2 powerpoints in it (ppt's with images and videos totalling 270MB). So they are good at reducing the number of files they have, but not the amount of data they have.

It is not the profile size. it is the data. Staff will copy files and save in different places. The excuse they give is lack of time to go through the documents and delete them What I don't understand is all teachers cleans their physical folders then why there is no time for e-folders?

[mavhc]

Deduplication is what you want then. Just don't run it on sparse VM disks, I did that once, it made the disk grow instead of release space, filled the physical disk and broke everything