Setting NTFS security rights without scanning / applying to every single sub file

[googlemad]

I have a 15TB (Yes TERABYTE!) archive for one of our systems, medical data so it has to be kept for 99999 years

 

The problem is it is comprised of millions and millions of sub folders and tiny files

 

I have now moved the data from a standalone SAN to a Windows VM but need to set the NTFS security permissions up...

 

Is there a way to set them in such a way at the root folder that sub folders and files will inherit the security settings from root without scanning / applying to every single sub file as Windows normally seems to do? Otherwise it could take weeks, even months with the amount of files within it!

[mavhc]

Probably not, ACLs are per file. I'd set group ACLs and then you can change the group members.

 

Probably a better way of doing it via another program rather than Explorer though.

 

Should have used ZFS though, how are you avoiding bitrot?

[willtech]

https://helgeklein.com/setacl-studio/ Might help you Edited May 19, 2020 by willtech

[psydii]

Indeed 'inheritance' of NTFS permissions was a very much mis-sold capability. If you are at a uni handling this data set, central IT might have a Microsoft Premier support contract which you can utilise to help solve this.

 

Given there isn't much you can do about the time it will take to enumerate and update 1M's of files, the next challenge is the restartablity of the process - if you hit a snag deploying the permissions how do you track where you were when you restart. I'm pretty sure I read a blogpost about this from a PFE, but MS nuked the technet blogs and my google-fu is unable to locate it.

[mavhc]

tried archive.org search?

[HPlum78]

ALICE! :-p

[Arthur]

Probably a better way of doing it via another program rather than Explorer though.

+1. File Explorer would be horrific.

 

The command-line version of SetACL can configure permissions and log to a file. There's also the NTFSSecurity PowerShell module. Both can handle long file names and paths too.

Edited May 19, 2020 by Arthur

[mrwoberts]

Would Security Groups be more helpful in this context, and what groups are set at the root level at the moment?

 

For example, granting the group 'TheRWcrew' permission to the root is easier in the long-term than adding/removing individual users.

I know that doesn't easily solve your problem now, so I guess we go back to find out what groups already have root folder permissions.

[Arthur]

The problem is it is comprised of millions and millions of sub folders and tiny files. Otherwise it could take weeks, even months with the amount of files within it!

Is the VM containing the files stored on one or more SSDs? If not...

 

www.scan.co.uk/products/1536tb-micron-9300-pro-25-ssd-u2-pcie-gen3-x4-nvme-3500mb-s-read-3500mb-s-write-850k-150k-iops

 

It would be a bit quicker then!

Edited May 19, 2020 by Arthur

[googlemad]

Thanks for the suggestions, apparently the supplier has a way to move the data and check integrity at the same time...we'll see! (I checked with TreeSize, for anyone wanting to know...6,223,156 files in 184,162 folders!)

[HPlum78]

Ah small fry 15TB, we have a couple of PB of that kind of data kicking about! But like you would not want to be Re-ACLing any of it! Also the prospect of getting it wrong would send our compliance and continuity manager into a full on melt down. Good luck.

[googlemad]

Just using Roadkill Unstoppable Copier to copy the data across to a new 16KB cluster sized volume (as we found the existing one was 4KB so couldn't extend past 15TB in the future!) and it is absolutely firing through it

 

The advantage is I set up the same named root folder on the new volume and set up my required permissions, now when it is copying the data into there it automatically inherits the correct permissions I have set so win win

[psydii]

Going way of the orignial topic, but 16Kb cluster size would likely waste a lot of space in a scenario where you were storing millions of small (<16Kb) files. But if they are *really* tiny they get stored in the MFT - and (this used to be true in 2000/2003 era) optimization of that needs to take place for volumes that will be handling a large number of files.

 

Finally I recall reading, but cannot find the reference, that on 4K disks NTFS will force a 4Kb cluster size regardless of the cluster size specified by the admin.

 

https://stackoverflow.com/questions/11619921/can-a-short-file-uses-more-mft-records-than-a-longer-file

https://docs.microsoft.com/en-us/windows/win32/fileio/master-file-table

https://support.microsoft.com/en-ca/help/174619/how-ntfs-reserves-space-for-its-master-file-table-mftv