Why is Dedup folder so big?

[jmak]

I'm trying to minimise the amount of data I back up. Using TreeSize to analyse the storage usage, it looks like I have a Staff directory with a size of 23GB which has 365.6 MB allocated. Presumably the difference is what appears in the ChunkStore in the System Volume Information? Is there any way that it could be bigger than that? The total Size shown in TreeSize for the directories I want to back up is 72.9GB, but the ChunkStore is 107GB. Could this relate to VSS?

 

Also WSUS Dedup (excluded from backup) has a size exactly double the allocated space. Does that mean that I have two copies of everything that could be cleaned up?

 

Thanks

[mikes]

Not sure exactly how dedup works but I always had trouble running space analyzers on drives with dedup on, it seemed to confuse them. I think I remember it counting more storage taken up than the drive volume actually contained.

I'm not sure why dedup would have two copies of everything in WSUS, I would think it would be smarter than that, is VSS on for your WSUS? I probably wouldn't have Dedup enabled on my WSUS drive anyway as I don't think any of the WSUS packages are identical?

 

You can run a dedup cycle by doing start-dedupjob, I always found I needed to do that after trying to free up space to see what I'd actually managed. I think if you do get-dedupjob (or a similar PS command) it will tell you exactly what it is doing?

[psydii]

Check that you have a garbage collection scheduled.

 

https://kallesplayground.wordpress.com/useful-stuff/windows-server-2012-r2-data-deduplication/

 

Also, unless you have thoroughly researched and tested your DR, excluding the dedupe folders from the backup is likely to ruin your day when you try to restore service after a failure.

[jmak]

Check that you have a garbage collection scheduled.

 

https://kallesplayground.wordpress.com/useful-stuff/windows-server-2012-r2-data-deduplication/

 

Also, unless you have thoroughly researched and tested your DR, excluding the dedupe folders from the backup is likely to ruin your day when you try to restore service after a failure.

100% never deleting or excluding the de-dup folders from back up! It's definitely where all the data I need is; the actual directories are thousands of times smaller, so it's not compressed data, it's only placeholders pointing to the Sys Vol folder. The reason for my query is whether it's expected that it's bigger than the reported size of the original directories or whether something's gone wrong. It doesn't seem logical to me, unless it's versioning, but that could be my faulty logic or that a developer had a moment of madness or there could be a very good reason that I don't know about.

 

I'll definitely have a look at garbage collection.

[ChrisMiles]

If you're doing file level backups you shouldn't need to back up the ChunkStore folder anyway though. The act of reading a file has nothing to do with dedupe, that all happens in the background. The size on disk is irrelevant. Not sure exactly what your setup is but I'd strongly recommend switching to block level storage backups when using dedupe. It preserves the deduplication and is much safer. You start messing about a deduped volume and you're going to have a very bad day one day.

[mikemcsharry]

This has been causing me a lot of issues as well. I work with two schools, both have / had dedup on for their data folders. In one case I'm replacing the server, in the other I'm adding a new box to take some file server roles.

Transferring / copying / backing up data has become a lottery. What you see seems to be not what you get.

garbagecollect goes some way to helping.

I've seen work by an MVP called Nick Whittome for some years, so when I found that he'd put an answer on his website it was worth a go. It's worth reading the page and also the comment someone has left regarding using the event logs (yep, there's a dedup event log down there) to check on progress.

A word of warning - I've been told that dedup plus encrypted disks is a very poor partnership...

https://nickwhittome.com/2014/10/01/disabling-data-deduplication-on-windows-server-2012r2/

[mikemcsharry]

I've followed Nick's advice (see above) on both networks I work with. Worked fine on both. Getting rid of dedup on a 500GB store took nearly 3 hours with nothing else running on the server.

The way to find out how the deduplication job has gone / is going is to go to event viewer and select applications and services, then select Microsoft and then windows and then deduplication. There are 3 logs, the activity log will show you that the job has queued, started and then finished, it will then log 'idle time out' after the job finishes.

 

The biggest problem with dedup is that it makes it impossible to see the wood for the trees.

The tiny user folder area and the massive system volume\chunks folder is not helpful. It doesn't guide you into which of your folders really are taking up a huge space. After removing data deduplication I found a 5GB recycle bin in the effected volume.

In answer to the very first question regarding size of the deduplication folders, this may help.. in server manager, select local server, then wait for the menu to populate (it seems to take a few seconds), then select file management and you'll see your drives. Select volumes, right click on the volume you're interested in and select de-duplication status from there. You'll see more information.

I've written the above note from memory and I'll check the exact paths and labels etc later today.