Printable View
This isn't a *nix vs MS thread. I'm capable in both, thanks. My network is a windows network and for the large part that's how it's staying. We do have *nix servers and we're happy with them.
So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.
Could you go into a *lot* more detail please?
I worry about anything that adds complexity to the backup. My ideal is a tape who's contents I can read and restore to virtually any type of file system with only the tape and a tape drive plugged into the computer. I preferably want several copies at any one time in several locations. That seems philosophically the opposite of de-duplication, perhaps not incompatible in the end but it certainly makes me cautious.
De duplication can work at a number of levels from file down to sub block. Maybe your organisation is storing "Company Name" in every office file as part of the properties. A de-duplication algorithm replaces that with a short token and then stores the the main data once and the token thousands of times. You then have thousands of files that are dependent for their integrity on one small piece of tape - a critical single point of failure.
I'm probably overly paranoid - but that's not IMO such a terrible sin when it comes to backups.
See this is where the information I have is lacking. I was under the impression that when doing this at file level that wouldn't be an issue. Block level I can certainly, for the reasons you describe @pcstru - I'm certainly more interested in file-level though. Is there anything in file-level dedupe that we should be cautious of?
10 people have the same 1GB file in their home folder, it's deduplicated so it uses 1GB plus the dedupe filesystem metadata.Quote:
So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.
Could you go into a *lot* more detail please?
Now use robocopy to copy to another filesystem. You now have 10GB of data. This isn't to say that you cannot potentially deduplicate the backup.
For optimized backups that maintain deduplication, he backup method needs to use the Data Deduplication Backup/Restore API and the storage backend needs to have the deduplication VSS writer available or handle deduplication in some other fashion. If you are using something like robocopy or rsync, they won't have the ability to be aware the file was deduplicated in the first place, or to attempt to write it back in a deduplicated state. Something like Backup Exec writes to virtual media files so deduplication has to happen before the write, or the deduplication must be done at the block level. So I imagine for Backup Exec to work properly the backup agent and storage services will have to be updated. If you are using Windows Server Backup on Server 2012 I guess you will have no problems.
Dedupe has been used in backups forever, if you are that worried about the media failing then you are not testing it adequately with verification and are still using tapes in a disk and SSD world. If it concerns you, you are paranoid or not taking enough backups anyway.
If you have a D2D2T device, then you can fit more backups on the D2D part.Quote:
Originally Posted by synaesthesia
What do you suggest, pull tapes out the store every few weeks to check them?Quote:
Originally Posted by SYNACK
Yes.Quote:
Originally Posted by pcstru
If you don't test your backups you might as well not bother running them.
Ben
I might agree that recovery procedures should be tested from time to time, long term archival of tape relies on verification at the time of writing and appropriate environmental storage conditions. No one pulls all the tapes and checks them every few weeks and doing so would severely reduce the tape life.
Veeam does dedupe on it's backups (done at VM) level, wonder how that will work with a 2012 file server VM...
Dedupe just works by linking stuff that is the same to the same place in the behind the scenes bit of the file system. When it is read out it reads out as a stream including the 'multiple' files as it is transparent to the user or unaware client programs. It would then be deduped again by Veeam. No big mystery.
I think we get around 35% space back with dedup on our NetApp :)
We're a secondary school with 1200 pupils and around 150 staff, just a figure to work off for you.
Attachment 15792
here is my 2012 server that hosts basically the student and staff shared area plus the network programs share. i've not got enough virtual servers on my 2012 server yet to get a good idea of the Dedupe rates there yet.
Very impressed with this feature. It is very easy to set up and has almost doubled the space available on my student drive. We have a large amount of data, and it was approaching 2TB. After running deduplication it is now around 1TB!
Attachment 15837
Was impressed when this was on the feature list for 2012 and even more so from the feedback above, well done MS :)
