Data De-duplication

[synaesthesia]

Anyone had a play with this feature yet? Reading up about it, it seems very useful - very, very much so on the average school network, I'd imagine. Would like to see some real-life figures. Not something we'd be considering for a while with a stock 2008R2 network (although we've a CCTV server that's already 2012'd) but could well be something written into the specs for future updates.

[SYNACK]

I'm looking at this as a direct driving force for the upgrade of our main storage server, I've looked, 45% of our usage at one site is thick-witted teachers copying rubbish from the central teachers drives to their own documents folders. Training these sheep to understand is not really an option short of brain augmentation so 2012 could actually easily save us a storage server upgrade for two to three whole years. That is quite a boost to our budget from a single bullet point feature of an OS we have available effectively for free through SA.

[morganw]

If your backups aren't de-duplicated then you could cause yourself a whole new problem...

[synaesthesia]

How so? The backups would therefore only back up the content once and the links thereafter surely?

[CyberNerd]

If your backups aren't de-duplicated then you could cause yourself a whole new problem...

 

Backups is usually the primary reason fordedupe especially if you can dedupe over the network as with solaris bsd etc.

 

So how come the interest in dedupe now? Its been in windows serving networks for years.. is it fanboism with ms or just love of a pretty metro interface?

[Arthur]

So how come the interest in dedupe now? It's been in Windows serving networks for years..

Because previously you would have had to buy a NAS running Windows Storage Server. Now it is included for "free" in Windows Server 2012.

[CyberNerd]

Because previously you would have had to buy a NAS running Windows Storage Server. Now it is included for "free" in Windows Server 2012.

 

Not at all. Its been in commercial solaris for years. Many solaris environments are there to serve windows networks. BSD has it and its even available for linux albeit with fewer features. So why the interest now? If not Fanboism or pretty interfaces, is it just down to a lack of knowledge in the people making decisions?

[plexer]

I guess lots of places may be windows only servers with local storage and now dedupe can be leveraged natively rather than via 3rd party hard/software.

 

Ben

[CyberNerd]

I guess, to those in the know, that you get the features that you want in your network for a cost effective price rather than what your current vendors have to offer.

[Arthur]

It's been in commercial Solaris for years.

You are correct that it has been able in other OSs for years, but in the context of this thread I was referring only to Windows-based OSs.

[plexer]

I guess, to those in the know, that you get the features that you want in your network for a cost effective price rather than what your current vendors have to offer.

 

Adding a windows 2012 file server at edu pricing would probably be more cost effective than anything else.

 

Ben

[morganw]

How so? The backups would therefore only back up the content once and the links thereafter surely?

 

It depends what you are using to do the backup. Even with software like Backup Exec I think deduplication of the backup requires an extra license, so you could end up inflating your data as you back it up.

[CyberNerd]

Adding a windows 2012 file server at edu pricing would probably be more cost effective than anything else.

 

Ben

 

If you are windows trained then yes. If your nix trained then no.

 

A bit of education on nix systems goes a long way in improving services for users

[plexer]

Seeing as the OP says he's running a windows based network then what I said holds.

 

Ben

[SYNACK]

If you are windows trained then yes. If your nix trained then no.

 

A bit of education on nix systems goes a long way in improving services for users

 

But then you get to deal with other hassles like downgrading client security for a couple of years till SMB2 was supported properly, slowdowns thanks to TCP window shaping and you get to deal with the whole thing all over again with SMB3, yay.

 

You can do it on Linux, you can do it on some storage boxes and have been able to for some time, you can even hack it together using WIM storage images if you really want but none of those are a substitute for a nice natively integrated feature that requires no extra stuffing about, platform collecting and overhead.

[synaesthesia]

This isn't a *nix vs MS thread. I'm capable in both, thanks. My network is a windows network and for the large part that's how it's staying. We do have *nix servers and we're happy with them.

 

So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.

Could you go into a *lot* more detail please?

[pcstru]

I worry about anything that adds complexity to the backup. My ideal is a tape who's contents I can read and restore to virtually any type of file system with only the tape and a tape drive plugged into the computer. I preferably want several copies at any one time in several locations. That seems philosophically the opposite of de-duplication, perhaps not incompatible in the end but it certainly makes me cautious.

 

De duplication can work at a number of levels from file down to sub block. Maybe your organisation is storing "Company Name" in every office file as part of the properties. A de-duplication algorithm replaces that with a short token and then stores the the main data once and the token thousands of times. You then have thousands of files that are dependent for their integrity on one small piece of tape - a critical single point of failure.

 

I'm probably overly paranoid - but that's not IMO such a terrible sin when it comes to backups.

[synaesthesia]

See this is where the information I have is lacking. I was under the impression that when doing this at file level that wouldn't be an issue. Block level I can certainly, for the reasons you describe @pcstru - I'm certainly more interested in file-level though. Is there anything in file-level dedupe that we should be cautious of?

[morganw]

So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.

Could you go into a *lot* more detail please?

 

10 people have the same 1GB file in their home folder, it's deduplicated so it uses 1GB plus the dedupe filesystem metadata.

Now use robocopy to copy to another filesystem. You now have 10GB of data. This isn't to say that you cannot potentially deduplicate the backup.

 

For optimized backups that maintain deduplication, he backup method needs to use the Data Deduplication Backup/Restore API and the storage backend needs to have the deduplication VSS writer available or handle deduplication in some other fashion. If you are using something like robocopy or rsync, they won't have the ability to be aware the file was deduplicated in the first place, or to attempt to write it back in a deduplicated state. Something like Backup Exec writes to virtual media files so deduplication has to happen before the write, or the deduplication must be done at the block level. So I imagine for Backup Exec to work properly the backup agent and storage services will have to be updated. If you are using Windows Server Backup on Server 2012 I guess you will have no problems.

[SYNACK]

Dedupe has been used in backups forever, if you are that worried about the media failing then you are not testing it adequately with verification and are still using tapes in a disk and SSD world. If it concerns you, you are paranoid or not taking enough backups anyway.

[Arthur]

Why would you dedupe the backup?

If you have a D2D2T device, then you can fit more backups on the D2D part.

[pcstru]

if you are that worried about the media failing then you are not testing it adequately with verification

 

What do you suggest, pull tapes out the store every few weeks to check them?

[plexer]

What do you suggest, pull tapes out the store every few weeks to check them?

 

Yes.

 

If you don't test your backups you might as well not bother running them.

 

Ben

[pcstru]

I might agree that recovery procedures should be tested from time to time, long term archival of tape relies on verification at the time of writing and appropriate environmental storage conditions. No one pulls all the tapes and checks them every few weeks and doing so would severely reduce the tape life.

[gshaw]

Veeam does dedupe on it's backups (done at VM) level, wonder how that will work with a 2012 file server VM...