Volume Shadow Copies - Used Space Isues & Confiusion

[CHiLL]

We've been slowly eating away at our staff file server's data drive and I'm perplexed as to why. At first, I thought someone had dumped their removable drive as a backup, or duplicated a large folder to the home drive. I also deleted the archived home folders of staff who left, since they're backed up to our long term storage. This was about 300GB and after deleting it, I saw no dent in the used space. This led me to question VSS, but looking at it, it's currently only using 33GB of space (presumably because the drive is >90% full).

 

I used Tree Size (run as Administrator) and I'm confused by it's readings. It says that the E:\ drive is size 5.9TB with 1.9TB allocated. It's a 2TB drive within Vmware, so I don't understand how it's reporting 6TB. Expanding the folders leads me to 1.8TB in E:\System Volume Information\Dedup. Research suggests this is where the VSS copies live - but looking at the files, a lot of them are super old, dating as far back as 2015! I believe this is where our space is being consumed - all these old files that haven't been deleted.

 

My question is - how do I delete these files? Is it possible to clean up these files without breaking the current snapshots?

Edited October 1, 2021 by CHiLL

[HPlum78]

Run the following:

vssadmin list shadowstorage

 

Post back the out please..

[CHiLL]

Run the following:

vssadmin list shadowstorage

 

Post back the out please..

PS C:\WINDOWS\system32> vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Shadow Copy Storage association
  For volume: (E:)\\?\Volume{73cbb39b-ddfc-4343-8983-62c0f935b347}\
  Shadow Copy Storage volume: (E:)\\?\Volume{73cbb39b-ddfc-4343-8983-62c0f935b347}\
  Used Shadow Copy Storage space: 26.2 GB (1%)
  Allocated Shadow Copy Storage space: 33.7 GB (1%)
  Maximum Shadow Copy Storage space: 205 GB (9%)

Shadow Copy Storage association
  For volume: (C:)\\?\Volume{c6fef8e2-0d30-11e5-80b5-806e6f6e6963}\
  Shadow Copy Storage volume: (C:)\\?\Volume{c6fef8e2-0d30-11e5-80b5-806e6f6e6963}\
  Used Shadow Copy Storage space: 0 bytes (0%)
  Allocated Shadow Copy Storage space: 0 bytes (0%)
  Maximum Shadow Copy Storage space: 5.96 GB (5%)

[mavhc]

Virtual file systems, like VSS, OneDrive, Google Drive, can show files as being size X but only taking up size Y (because they're clones of another file in VSS's case, or not actually on the computer in OneDrive/Google Drive's case)

[CHiLL]

I'm just seeing a bunch of these files in TreeSize:

[ATTACH=CONFIG]63257[/ATTACH]

They're not linked to existing VSS snapshots according to VSS. How can I delete them? Can I litterally just delete them from within TreeSize, or do I have to use VSS commands? I'm not actually sure VSS will remove them if I disable, since it doesn't appear to know about them.

[mavhc]

https://community.spiceworks.com/topic/2108623-dedup-chunkstore-is-massive suggests you can turn off dedupe

[bald_pig]

It sounds like you have file deduplication enabled?

[CHiLL]

I thought dedupe was part of how VSS works? VSS and dedupe are things I don't have much experience in! Is dedupe something that's normally enabled on a file share, such as home and shared folders? If so, why is this causing me a problem?

 

Edit:

Looking at the Task Scheduler on the server, under Task Scheduler Library > Microsoft > Windows > Deduplication, I can see three tasks:

• BackgroundOptimization
  
• WeeklyGarbageCollection
  
• WeeklyScrubbing
  

BackroundOptimization is scheduled hourly and the other two are scheduled weekly. They all ran sucessfully on their last run apparently (14:45 today, and last Saturday respectively).

 

Edit 2:

Just ran these commands:

PS C:\WINDOWS\system32> Get-DedupStatus

FreeSpace    SavedSpace   OptimizedFiles     InPolicyFiles      Volume
---------    ----------   --------------     -------------      ------
119.06 GB    1.92 TB      1121528            1121458            E:


PS C:\WINDOWS\system32> Get-DedupVolume

Enabled            UsageType          SavedSpace           SavingsRate          Volume
-------            ---------          ----------           -----------          ------
True               Default            1.92 TB              50 %                 E:

So does that mean if dedupe is disabled, I'd need an extra 1.92TB attached to the E: drive to accomodate that space?

Edited October 1, 2021 by CHiLL

[bald_pig]

Dedupe is not a part of VSS. dedupe is short for deduplication, and is essentially a compression method. It is very common to use on storage servers, but if you have a lot of files that are already compressed, like videos and photos, then you can end up using more space.

 

Treesize doesn't understand dedupe, which is why you get bad readings. The size on disk will give you how much space is used on the disk, whereas using the regular size and temporarily excluding the dedup will give you the actual data size.

 

Just seen your edit, yes, you would need an extra 1.92tb on that drive to store everything without deduplication.

[mavhc]

Try WizTree to look at space, it shows space used and allocated

[bald_pig]

Try WizTree to look at space, it shows space used and allocated

 

So does treesize, which they're already using.

[Norphy]

Dedupe is not a part of VSS. dedupe is short for deduplication, and is essentially a compression method. It is very common to use on storage servers, but if you have a lot of files that are already compressed, like videos and photos, then you can end up using more space.

 

Treesize doesn't understand dedupe, which is why you get bad readings. The size on disk will give you how much space is used on the disk, whereas using the regular size and temporarily excluding the dedup will give you the actual data size.

 

Just seen your edit, yes, you would need an extra 1.92tb on that drive to store everything without deduplication.

 

Deduplication isn't a compression method. Deduplication removes duplicate copies of data on a storage device. So if someone puts on a file called "Super fun happy time.jpg" and then copies it 500 times, the 499 duplicates are removed and replaced with a pointer to the original file instead. Therefore, it doesn't matter whether a file is compressed or not, if there are 500 duplicates of a jpg file, the other 499 will still be removed and the space will be recovered. That said, most file systems do this at the block level rather than the file level these days.

 

Now, if you're enabling traditional on-the-fly compression on a folder and/or file system then yes, what you've said is right. You generally can't compress that that's already been compressed. If anything, the extra overhead of the additional compression format will make files larger.

 

/edit

 

it's worth mentioning that some of the tree sizing programs don't work properly on deduped volumes - There was a modified version of WinDirStat which enumerates duplicated disk space properly called WinDirStatDD

 

/edit 2

 

Reading the rest of the thread, you need a more aggressive garbage collection policy on your deduplicated volume. When you delete a file, the space won't be recovered immediately. Run the following PS command:

 

Start-DeDupJob -type GarbageCollection -volume d:

 

https://docs.microsoft.com/en-us/powershell/module/deduplication/start-dedupjob?view=windowsserver2019-ps

Edited October 1, 2021 by Norphy

[bald_pig]

Deduplication isn't a compression method. Deduplication removes duplicate copies of data on a storage device. So if someone puts on a file called "Super fun happy time.jpg" and then copies it 500 times, the 499 duplicates are removed and replaced with a pointer to the original file instead. Therefore, it doesn't matter whether a file is compressed or not, if there are 500 duplicates of a jpg file, the other 499 will still be removed and the space will be recovered. That said, most file systems do this at the block level rather than the file level these days.

 

Now, if you're enabling traditional on-the-fly compression on a folder and/or file system then yes, what you've said is right. You generally can't compress that that's already been compressed. If anything, the extra overhead of the additional compression format will make files larger.

 

Deduplication will split files into chunks, it doesn't just work on the file as a whole. Also, saying something is like compression, isn't saying it is compression.

 

From https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand#how-does-dedup-work (an interesting read into the processes behind the system)

• How does Data Deduplication differ from archive file formats like zip, rar, 7z, cab, etc.? Archive file formats, like zip, rar, 7z, cab, etc., perform compression over a specified set of files. Like Data Deduplication, duplicated patterns within files and duplicated patterns across files are optimized. However, you have to choose the files that you want to include in the archive. Access semantics are different, too. To access a specific file within the archive, you have to open the archive, select a specific file, and decompress that file for use. Data Deduplication operates transparently to users and administrators and requires no manual kick-off. Additionally, Data Deduplication preserves access semantics: optimized files appear unchanged after optimization.
  

[Norphy]

Deduplication will split files into chunks, it doesn't just work on the file as a whole. Also, saying something is like compression, isn't saying it is compression.

 

From https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand#how-does-dedup-work (an interesting read into the processes behind the system)

 

You said it was essentially compression, not that it was like compression. Even then, I'd still argue that it's not like compression, just that it's another method of using less disk space. And I said that it does it on the block level rather than the file level.

 

There are some dedup methods that work on the file level, such as Single Instance Store that Exchange Server used to use but they're more unusual these days.

[CHiLL]

/edit 2

 

Reading the rest of the thread, you need a more aggressive garbage collection policy on your deduplicated volume. When you delete a file, the space won't be recovered immediately. Run the following PS command:

 

Start-DeDupJob -type GarbageCollection -volume d:

 

https://docs.microsoft.com/en-us/powershell/module/deduplication/start-dedupjob?view=windowsserver2019-ps

I don't understand why I still have files dated 2015, 2016, 2017, etc...when the scheduled job to perform that task is running successfully every week.

Edited October 1, 2021 by CHiLL

[andy_b]

I still use this for dedup'd FS:

 

http://www.edugeek.net/forums/windows-server-2012/145138-tool-like-treesize-can-reflect-upon-true-file-size-when-deduplication-enabled.html

[HPlum78]

Due to being busy this has moved on a bit and we have gone past my next command of Get-DedupStatus..

 

I will add my thoughts around deduplication and compression.

 

Deduplication works at the bock level and the algorithm is not concerned about the name of the files. It is looking for matching blocks in its data set, once found these blocks are removed and replaced with a reparse point to the required block stored in the deduplication chunk store. This is a post write process, for reads there is a storage filter driver that repoints read operations to the chunck store when and if a reparse point is hit.

 

Compression on the other hand is done at the binary level in a block, the algorithm eliminates redundant data and when decompression is required that same algorithm is used to rebuild the binary data. This can be done at a volume level in windows or by using your favourite compression utility.

 

Using compression and deduplication on an NTFS volume will mash your data because of the way the post processing of the data happenes and how the CRC is calculated when using volume compression. That said in ReFS drives on Windows 2019 and later now support the use of both deduplication and compression (could be 2016 though just cannot remember)

 

So to answer the question deduplication saves space by removing duplicate blocks of data and storing those in a central location to be rebuilt at read time. Compression removes redundant binary data and uses an algorithm to decompress and rebuild the data at read time.

Edited October 2, 2021 by HPlum78

[CHiLL]

So to update this, over the weekend I ran the following command:

Start-DedupJob -type GarbageCollection -Volume E: -Full

It cleaned up about 80GB, but I'm still using 1.8TB in the chunk store, with many files dating back to 2015 (when the server was created). How do I remove these?

[HPlum78]

So those files in the chunck store even if they was created in 2015 when the server was built/ deduplication was configured are more than likely still required as they will have reparse pints associated with them. If that makes sense....

[bald_pig]

They are probably chunks of system files. Looking at the numbers you previously posted, your chunk store size matches up with what I'd expect it to be. if you have 1.9Tb of savings with 50% deduplication, you'd expect around a 1.9tb chunk store.

[mavhc]

How do you get a list of files from 2015 that are in the chunk store?

[CHiLL]

Ah, I see. Good job I didn't just delete them!

 

I've used WinDirStatDD to break down the files too and I'm seeing that our media guy has some 300GB+ of videos that we can store on SharePoint instead of locally. Plus a load of other videos, like BTEC Sport evidence. I need to liase with the PE department to move the older files to SharePoint too (we also have these all backed up in our EoY backups too), so the data is never gone.

[Norphy]

Ah, I see. Good job I didn't just delete them!

 

I've used WinDirStatDD to break down the files too and I'm seeing that our media guy has some 300GB+ of videos that we can store on SharePoint instead of locally. Plus a load of other videos, like BTEC Sport evidence. I need to liase with the PE department to move the older files to SharePoint too (we also have these all backed up in our EoY backups too), so the data is never gone.

 

I've had the same conversations with depressing regularity with a lot of people. One time, I even had to take it to SMT because a lot of morons had downloaded pirated films stored in the shared areas. How did I know they were pirated? They still had the bloody torrent name in the filename of the video.

 

You may need to get SMT backing for this kind of thing to get it enforced, otherwise you'll be shouting at people again in three months time.

[mavhc]

Tell them the auto piracy scanner auto deleted them and auto sent their name and address to the authorities.

[Norphy]

Tell them the auto piracy scanner auto deleted them and auto sent their name and address to the authorities.

 

The pirated videos were deleted and the owners of the files were disciplined. I kept an eye out for about a year afterwards and didn't see any repeat offensives, happily enough.

 

When I'm talking about getting SMT backing, I'm not talking about removing illegal content, I'm talking about moving the types of content that CHiLL is talking about to a more suitable area and to enforce making the more suitable area the default place to store stuff like that. Otherwise it'll keep on coming back.