Deduplication - anyone else using it?

[Dave_O]

25% is about right for user files according to Microsoft's guides. Up to 50% for shared areas. 75% for software deployment areas and 90% for VHDs.

 

Checked the machine two weeks later and the 25% has gone up to 38%. Does it improve over time?

[localzuk]

The process is done via a pair of scheduled tasks each night. I believe they have a limit of how long they run for each night also. So, yes, the values can go up over time as it optimises more and more.

 

I have come across a slight issue with backups. Backing up using BackupAssist seems to have issues with the hard links that are used, and the backups often just end up as a bunch of 4k files rather than the actual files.

[FN-GM]

Hi,

 

I have enabled this in production now. But when i look at the disks in Server Manager i can see a deduplication rate of 30% i right click and refresh it changes to say 28%. I refresh it again and it goes back up to 31%. Why is it so erratic, changing the figures in seconds? Will it settle down?

 

Thanks

 

This seems to have stopped overnight.

[Ephelyon]

Our Veeam setup makes great use of dedupe for the backups. I forget the ratio now but it helps a lot, not just for storage but also (because this is source dedupe) in terms of how much data we have to transfer over the wire for our offsite backups to another school a few miles up the road.

[zbjsy]

Just to give you some quick results from some fileservers - just because some beat the 28%. Need to do some CPD on the need not to copy from the shared area to my documents! Obviously big advantages up front - has anyone come across any downsides?

 

Processed files: 331578

Processed files size: 539.95 GB

Optimized files size: 328.72 GB

Space savings: 211.17 GB

Space savings percent: 39

 

Processed files: 117387

Processed files size: 200.52 GB

Optimized files size: 126.07 GB

Space savings: 74.43 GB

Space savings percent: 37

 

Processed files: 150567

Processed files size: 298.56 GB

Optimized files size: 223.50 GB

Space savings: 75.05 GB

Space savings percent: 25

 

what command did you run to get that info? Also is there a way to list the most duplicated files?

[Geoff]

I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful. You need to look at your workloads and check if dedupe is really worth it to you. In my case the system overhead caused by running deduping was not worth the performance hit. So when hard drives are so cheap per GB in the long run I found it better overall not to dedupe and just throw more disk space at the problem.

[john]

I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful. You need to look at your workloads and check if dedupe is really worth it to you. In my case the system overhead caused by running deduping was not worth the performance hit. So when hard drives are so cheap per GB in the long run I found it better overall not to dedupe and just throw more disk space at the problem.

 

Would agree with Geoff on this the overhead for de-dupe on the Sun / Oracle SANs were not worth it for us, although the space saved would be nice the loss of performance was not acceptable so it was turned back off.

[Geoff]

I only really use it on our (120Tb) archive system currently. It is rarely accessed and using space efficiently is more important than speed of access. This system also runs with compression on some parts of the file system although this is rapidly becoming an irrelevance as file formats start to adopt compression natively (wav -> mp3 and office 97 -> office XML). Our day to day working file set (6Tb) runs uncompressed and with no deduplication.

[Arthur]

I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful.

Dedupe on Server 2012 works in a totally different way compared to ZFS. The performance hit is extremely minimal* and you don't need huge amounts of RAM. Read the TechNet links below for details.

 

• Introduction to Data Deduplication in Windows Server 2012 « The Storage Team Blog
  
• Deduplication Cmdlets in Windows PowerShell
  

* Assuming you don't do something stupid like setting MinimumFileAgeDays to 0 which forces Windows to constantly dedupe the volume(s) that have it enabled.

 

For volumes with a lot of redundant data, the space savings can be massive.

 

http://i.imgur.com/FR6uyPH.png

 

@Geoff. I'm guessing your Oracle SAN was under-specced for the amount of data you were deduping? From what I have read, 1TB of deduplicated data (4KB average block size) would require around 80GB RAM to hold the dedupe tables and need a beast of a CPU to do the SHA256 checksum calculations.

Edited May 31, 2013 by Arthur

[FN-GM]

Got it in production here.

 

33% on file server

55% on VLE server.

 

No performance hits either.

[psydii]

Long long ago I co-hosted my user data with the Windows 2000 RIS service.

 

This meant i got de-dupe on my user shares. It wasn't entirely painless, and about 6 months after I first encountered a non-critical problem Microsoft added a note to say that it was not a recommended configuration. I never lost data, but restores could be 'fun'. It gave me about 40% more space that I had paid for and kept backup sizes down.

 

I'm intrigued to see it come back into the Windows File Server role 14 years after it's initial outing. Looks like they've done a lot of work on it. Data Deduplication Overview

 

In this incarnation I'd be surprised if it isn't a by-product of work done making Azure VMs space efficient.

Edited May 31, 2013 by psydii

[Geoff]

@Geoff. I'm guessing your Oracle SAN was under-specced for the amount of data you were deduping? From what I have read, 1TB of deduplicated data (4KB average block size) would require around 80GB RAM to hold the dedupe tables and need a beast of a CPU to do the SHA256 checksum calculations.

 

It's not an Oracle SAN. However the front end server has 128Gb of RAM and Intel Xeons with 4x4 Cores. It also has a hardware card to offload the SHA256. So it's not like I didn't try. However 2012's dedupe runs on a schedule does it not? Because we're a 24/7 operation so I would need it to run real-time. Regardless, I just gave up on the idea and threw more storage at the problem.

[ADMaster]

I have deduplication enabled on the data drive for my win8 box at home. I see between 55 – 60 % savings.

 

I will be getting Server 2012 later this summer and plan to move all my shares to it. The eval tool says I’ll get 58% on my WDS / Software share and 27% on my user data share.

@zbjsy

From an elevated PowerShell session these commands will get you the info you’re looking for.

 

Get-DedupStatus

Get-DedupVolume

 

Deduplication Cmdlets in Windows PowerShell

[FN-GM]

However 2012's dedupe runs on a schedule does it not?

 

It can do and/or it can run in the background. When the server is under load it will pause dedupe automatically.

Edited May 31, 2013 by FN-GM

[localzuk]

Just moved our staff files over to our 2012 server, and deduplication has run on the data. Total saving of 47% or 478GB.