Deduplication - anyone else using it?

[localzuk]

We've been transferring all our data from a Windows Server 2003 machine over to our new Windows Server 2012 machine and one of the features I'm loving is dedupe, and how simple it is to use.

 

Anyone else using deduplication on their servers now? Not necessarily with 2012.

 

What sort of savings are you seeing? We're hitting a 25% deduplication rate here. Which is pretty good, considering we've not moved everything over to that server yet either.

[british_government]

We are looking at transferring over summer and using it. I ran the analysis tool on our existing data and we were seeing similar results, around 28%.

 

One of my queries on it has been backup, if your original data size is 100gb, and deduplicated means its 75gb, what is your backup size? Presumably 100gb?

[localzuk]

The backup size depends on the tools you use for backup. If they support Server 2012, they should be able to use the optimised backup option, which would mean your backup would be 75GB. However, if they don't support it, it would be 100GB.

 

The technical(ish) details of it are here - Backup and Restore of Data Deduplication-Enabled Volumes (Windows)

[british_government]

Ah great thanks, we are looking at implementing a new backup system too that would be DPM 2012 so in that case our backup sizes should be reduced too. Obviously with incremental backups it gets a bit confusing and things I don't even want to think about!

[FN-GM]

I took all our file server on a test box and enabled it and we where saving 30% ish. We use DPM 2012 SP1 for backups, but i didnt test the backups to be honest.

[sonofsanta]

Not live with it yet, but on our data drive the tool told us 39% reduction in space without compression, which is phenomenal, and almost worth the aggro of having to work with Server 2012. Our roaming profile data was more than halved without compression, though I wonder if it's a net benefit (less data) or net harm (more file work?) to the SSDs. Probably unlikely to massively impact either way, tbh, and the SSDs have ample endurance to cope anyway.

 

You can't make me use 2012 anywhere else though, no you can't. (Though I'd probably use it for virtual hosts if I was virtualising now, but I'm not messing with my existing setup)

[FN-GM]

You can't make me use 2012 anywhere else though, no you can't.

@sonofsanta How do you mean?

[localzuk]

Server 2012 is a far superior server OS than 2003, 2008 or 2008 R2. It is faster, has a far improved I/O system, better network performance etc... Don't know why you wouldn't want to use it!

[sonofsanta]

But... but... the interface!

[localzuk]

I don't think I've been in the 'metro' bit for more than a few seconds at a time...

[sonofsanta]

I don't think I've been in the 'metro' bit for more than a few seconds at a time...

 

Even then, it seems to occasionally decide that no, you can't layer that particular combination of windows in such a way that you can keep an eye on them all at the same time. Maybe it's just because I've only ever dealt with it in fifteen minute bursts and never forced myself to actually work through the nausea, but still, the very notion of a touchscreen interface on a server OS makes me itch all over with horror.

[FN-GM]

To be honest everything should be done on remote consoles on a client. You should hardly ever need to log onto servers. I hardly ever log onto servers.

[sted]

But... but... the interface!

 

you can get to all you need from server manager ad/dns/dhcp without ever going into metro and metro sucks for about a fortnight then it starts to make sense imo

[RTFM]

We have sedum on our SAN, it's fantastic

[truebluesteve]

In regards to the backups of de duped drives, is anyone using them with cloud based backup systems which do their own de duping and how it affects backup sizes?

[pete]

But... but... the interface!

 

*cough* mmc *cough*

[sonofsanta]

I know, I know... really I should just get on with server core and do the job properly. In my more conspiratorial moments I have suspected that this is Microsoft's intention, and that server core itself is a plan to boost the numbers taking MS certification.

[gh5000]

Just to give you some quick results from some fileservers - just because some beat the 28%. Need to do some CPD on the need not to copy from the shared area to my documents! Obviously big advantages up front - has anyone come across any downsides?

 

Processed files: 331578

Processed files size: 539.95 GB

Optimized files size: 328.72 GB

Space savings: 211.17 GB

Space savings percent: 39

 

Processed files: 117387

Processed files size: 200.52 GB

Optimized files size: 126.07 GB

Space savings: 74.43 GB

Space savings percent: 37

 

Processed files: 150567

Processed files size: 298.56 GB

Optimized files size: 223.50 GB

Space savings: 75.05 GB

Space savings percent: 25

[british_government]

I'm not sure if it is a downside or not, we are looking at putting all our file shares on one drive on one server to get the most out of deduplication. We run a hyper-v environment so really this shouldn't have any performance impact compared to on 2 different virtual servers and 4 different virtual drives at the moment.

 

Just feels like adding a single point of failure, but really it isn't I guess!

[pete]

Just poked around on one of ours (2008R2, running SIS) and it's quite effective.

 

[localzuk]

There is a downside to deduplication, and to SIS. If a sector becomes corrupt for whatever reason, and it happens to be a file (SIS) or 'chunk' of a file (Deduplication) that has multiple copies across the disk, you end up losing all those copies.

 

But then, we all have backups!

[Dave_O]

We're hitting a 25% deduplication rate here. Which is pretty good, considering we've not moved everything over to that server yet either.

 

Got 24% on a 1.5Tb fileserver, was hoping for more. Obviously staff don't copy the things into different folders as much as I thought they did

[localzuk]

25% is about right for user files according to Microsoft's guides. Up to 50% for shared areas. 75% for software deployment areas and 90% for VHDs.

[FN-GM]

Hi,

 

I have enabled this in production now. But when i look at the disks in Server Manager i can see a deduplication rate of 30% i right click and refresh it changes to say 28%. I refresh it again and it goes back up to 31%. Why is it so erratic, changing the figures in seconds? Will it settle down?

 

Thanks

[ass17]

We have sedum on our SAN, it's fantastic

 

Tell me more... Sedum