Data De-duplication

[SYNACK]

Dedupe just works by linking stuff that is the same to the same place in the behind the scenes bit of the file system. When it is read out it reads out as a stream including the 'multiple' files as it is transparent to the user or unaware client programs. It would then be deduped again by Veeam. No big mystery.

[RTFM]

I think we get around 35% space back with dedup on our NetApp

 

We're a secondary school with 1200 pupils and around 150 staff, just a figure to work off for you.

[k-strider]

here is my 2012 server that hosts basically the student and staff shared area plus the network programs share. i've not got enough virtual servers on my 2012 server yet to get a good idea of the Dedupe rates there yet.

[mitchell1981]

Very impressed with this feature. It is very easy to set up and has almost doubled the space available on my student drive. We have a large amount of data, and it was approaching 2TB. After running deduplication it is now around 1TB!

 

[gshaw]

Was impressed when this was on the feature list for 2012 and even more so from the feedback above, well done MS

[AnthonySomaro]

Anyone had a play with this feature yet? Reading up about it, it seems very useful - very, very much so on the average school network, I'd imagine. Would like to see some real-life figures. Not something we'd be considering for a while with a stock 2008R2 network (although we've a CCTV server that's already 2012'd) but could well be something written into the specs for future updates.

 

I think this article (though a little long) might interest you, because many real-world stats are shown:

Happy SysAdm: Real world data deduplication savings under WIndows 2012

 

Anthony

[Alis_Klar]

The policy a year or two ago in the Exchange division was to drop the dedupe (single instance storage dropped in Exchange 2010) features of Exchange as storage is "so cheap" these days. Couldn't understand this at all as when people send the same attachment to 30 people this has obvious dedupe potential.

 

http://blogs.technet.com/b/exchange/archive/2010/02/22/dude-where-s-my-single-instance.aspx

 

Pleasantly surprised the Server OS division are pushing dedupe again. I wonder if the exchange division will now look at dedupe again.

Edited February 15, 2013 by Alis_Klar

[SYNACK]

The policy a year or two ago in the Exchange division was to drop the dedupe (single instance storage dropped in Exchange 2010) features of Exchange as storage is "so cheap" these days. Couldn't understand this at all as when people send the same attachment to 30 people this has obvious dedupe potential.

 

http://blogs.technet.com/b/exchange/archive/2010/02/22/dude-where-s-my-single-instance.aspx

 

Pleasantly surprised the Server OS division are pushing dedupe again. I wonder if the exchange division will now look at dedupe again.

 

Exchange dropped it as their method was resource heavy, suprise, suprise from the Exchange team. They might pick it up if the server os or more importantly the sql team do a better job of it.

[DEvans]

Most backup software suppliers are making their software Windows Data Dedupe aware, so this will come with time. For now, you have few options. The top three, of which I have used recently during the R&D Process of Server 2012 are:

 

Backup Exec 2012 - Expensive because you need the full edition (not OEM) and the data dedupe awareness plugin, which comes at another cost. When working though, it is brilliant, if you get around the learning curve.

 

Data Protection Manager 2012 - Certainly the best product for backing up Microsoft Networks, since its made by them. You will need ISCSI/FC SAN like storage devices to use this to its full potential as standard NAS is not supported and it still doesn't support full grandfather like backup to tape/ISCSI/whatever without some for of expensive plugin, sorta hard to explain that with my simple mind, you know what i'm on about if you use it, its more of a live backup system, always backing up, every few minutes it syncs, very cool and extremely reliable and quick. Comes at a significant cost though, like all System Center Products, thats not even including the hardware to backup to. ISCSI is becoming cheap now, QNAP 12TB 1U is less than 1K now. Possibly a better choice than a true file server.

 

Inbuilt Windows Server Backup - The basic of basics and it's free, but it works surprisingly well, though you can only backup to a single location and so far Data Dedupe on or off doesn't cause any problems. (yes I have actually attempted restores, i don't work in theory here)

 

 

As for the Data Dedupe feature in itself, if you can get a copy of 2012, you can steal the data dedupe eval tool, which you can run on your file server and it will give you a fairly realistic idea on how much data you can save if you turned it on each volume.

 

I have the luxury of going to multiple sites and different schools and some of them have saved over 60% of their storage. !!

 

We are talking Gigabits to Terabytes (school sizes vary). So it does look like a fantastic feature for schools on a budget, especially your primaries and middles. (all those photos for example in nursery, thats a good few MB per photo saved).

 

Just make sure the backup you are using is data dedupe aware (usually given away by the phrase "Server 2012 Compatible") but always give your supplier a call to double check it's data dedupe capabilities and let all the complicated stuff be handled by those that are paid more than us to truely understand the deep down technology of Server 2012's Data Dedupe, I have a good understanding of it, but there still are a few, "how the hell does it figure that out?" moments occuring.

 

Hope this helps.

 

Kept on topic well there, this thread seemed to be venturing into the whole nix vs windows vs world conversations .. again.

 

 

Dan

 

The policy a year or two ago in the Exchange division was to drop the dedupe (single instance storage dropped in Exchange 2010) features of Exchange as storage is "so cheap" these days. Couldn't understand this at all as when people send the same attachment to 30 people this has obvious dedupe potential.

 

I completely agree that this seemed like a mental decision from the team.

 

However, let's see how Exchange 2013 (or Office 365) fairs. Exchange 2013 (as far as I know and at time of writing this) doesn't seem to have a proper migration path from any previous version of Exchange. It doesn't allow for the coexistence, yet.

 

I have a sneaking feeling that MS Exc are going down a very IBM'y route and the data will be stored in the database alone, mailboxes, emails etc, all in one place. Massively beneficial when it comes to upgrades.

 

Unmount Database, transfer database to new server, Mount database. Done. Most likely a live transfer to remove any disruption. Then of course, one database is easier to control and they can stick on some fancy technologies to data dedupe within that single database. I'm thinking out loud, but this is similar to how some other large companies manage their email systems, IBM being my mentioned example.

Edited February 26, 2013 by DEvans

[DrCheese]

Backup Exec 2012 - Expensive because you need the full edition (not OEM) and the data dedupe awareness plugin, which comes at another cost. When working though, it is brilliant, if you get around the learning curve.

Careful, BE2012 does not offer official support for 2012 yet, nor the dedupe function built into server 2012. The dedupe plugin your speaking of is Symantec's own, that does the same job but on backups. I would be scared of combining the two at present.

 

Symantec are doing a service pack for BE2010/2012 that will support this tho and it's out (in beta form at least) soon.

[DEvans]

Careful, BE2012 does not offer official support for 2012 yet, nor the dedupe function built into server 2012. The dedupe plugin your speaking of is Symantec's own, that does the same job but on backups. I would be scared of combining the two at present.

 

Symantec are doing a service pack for BE2010/2012 that will support this tho and it's out (in beta form at least) soon.

 

I didn't realise this until I went back and had a proper look.

 

Though the backups have been successful, I am working in a lab enviroment. Restoration so far has been fine with Windows Data Dedupe on, but with this information I obviously could never recommend Backup Exec as a live backup solution.

 

Thanks for this,

 

/Dan