+ Post New Thread
Page 2 of 3 FirstFirst 123 LastLast
Results 16 to 30 of 36
Windows Server 2012 Thread, Data De-duplication in Technical; This isn't a *nix vs MS thread. I'm capable in both, thanks. My network is a windows network and for ...
  1. #16

    synaesthesia's Avatar
    Join Date
    Jan 2009
    Location
    Northamptonshire
    Posts
    5,488
    Thank Post
    527
    Thanked 876 Times in 683 Posts
    Blog Entries
    15
    Rep Power
    438
    This isn't a *nix vs MS thread. I'm capable in both, thanks. My network is a windows network and for the large part that's how it's staying. We do have *nix servers and we're happy with them.

    So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.
    Could you go into a *lot* more detail please?

  2. #17


    Join Date
    May 2009
    Posts
    2,488
    Thank Post
    198
    Thanked 629 Times in 481 Posts
    Rep Power
    227
    I worry about anything that adds complexity to the backup. My ideal is a tape who's contents I can read and restore to virtually any type of file system with only the tape and a tape drive plugged into the computer. I preferably want several copies at any one time in several locations. That seems philosophically the opposite of de-duplication, perhaps not incompatible in the end but it certainly makes me cautious.

    De duplication can work at a number of levels from file down to sub block. Maybe your organisation is storing "Company Name" in every office file as part of the properties. A de-duplication algorithm replaces that with a short token and then stores the the main data once and the token thousands of times. You then have thousands of files that are dependent for their integrity on one small piece of tape - a critical single point of failure.

    I'm probably overly paranoid - but that's not IMO such a terrible sin when it comes to backups.

  3. #18

    synaesthesia's Avatar
    Join Date
    Jan 2009
    Location
    Northamptonshire
    Posts
    5,488
    Thank Post
    527
    Thanked 876 Times in 683 Posts
    Blog Entries
    15
    Rep Power
    438
    See this is where the information I have is lacking. I was under the impression that when doing this at file level that wouldn't be an issue. Block level I can certainly, for the reasons you describe @pcstru - I'm certainly more interested in file-level though. Is there anything in file-level dedupe that we should be cautious of?

  4. #19
    morganw's Avatar
    Join Date
    Apr 2009
    Location
    Cambridge
    Posts
    815
    Thank Post
    46
    Thanked 132 Times in 126 Posts
    Rep Power
    39
    So back on topic, @morganw I still don't understand what dedupe has to do with backups? Why would you dedupe the backup? One of the best reasons I can think of to use dedupe in the first place is shrinking backup size so it's already done.
    Could you go into a *lot* more detail please?
    10 people have the same 1GB file in their home folder, it's deduplicated so it uses 1GB plus the dedupe filesystem metadata.
    Now use robocopy to copy to another filesystem. You now have 10GB of data. This isn't to say that you cannot potentially deduplicate the backup.

    For optimized backups that maintain deduplication, he backup method needs to use the Data Deduplication Backup/Restore API and the storage backend needs to have the deduplication VSS writer available or handle deduplication in some other fashion. If you are using something like robocopy or rsync, they won't have the ability to be aware the file was deduplicated in the first place, or to attempt to write it back in a deduplicated state. Something like Backup Exec writes to virtual media files so deduplication has to happen before the write, or the deduplication must be done at the block level. So I imagine for Backup Exec to work properly the backup agent and storage services will have to be updated. If you are using Windows Server Backup on Server 2012 I guess you will have no problems.

  5. #20

    SYNACK's Avatar
    Join Date
    Oct 2007
    Posts
    10,692
    Thank Post
    824
    Thanked 2,570 Times in 2,187 Posts
    Blog Entries
    9
    Rep Power
    731
    Dedupe has been used in backups forever, if you are that worried about the media failing then you are not testing it adequately with verification and are still using tapes in a disk and SSD world. If it concerns you, you are paranoid or not taking enough backups anyway.

  6. #21


    Join Date
    Feb 2007
    Location
    51.405546, -0.510212
    Posts
    8,125
    Thank Post
    203
    Thanked 2,390 Times in 1,770 Posts
    Rep Power
    704
    Quote Originally Posted by synaesthesia View Post
    Why would you dedupe the backup?
    If you have a D2D2T device, then you can fit more backups on the D2D part.

  7. #22


    Join Date
    May 2009
    Posts
    2,488
    Thank Post
    198
    Thanked 629 Times in 481 Posts
    Rep Power
    227
    Quote Originally Posted by SYNACK View Post
    if you are that worried about the media failing then you are not testing it adequately with verification
    What do you suggest, pull tapes out the store every few weeks to check them?

  8. #23

    plexer's Avatar
    Join Date
    Dec 2005
    Location
    Norfolk
    Posts
    12,977
    Thank Post
    587
    Thanked 1,496 Times in 1,342 Posts
    Rep Power
    398
    Quote Originally Posted by pcstru View Post
    What do you suggest, pull tapes out the store every few weeks to check them?
    Yes.

    If you don't test your backups you might as well not bother running them.

    Ben

  9. #24


    Join Date
    May 2009
    Posts
    2,488
    Thank Post
    198
    Thanked 629 Times in 481 Posts
    Rep Power
    227
    I might agree that recovery procedures should be tested from time to time, long term archival of tape relies on verification at the time of writing and appropriate environmental storage conditions. No one pulls all the tapes and checks them every few weeks and doing so would severely reduce the tape life.

  10. #25
    gshaw's Avatar
    Join Date
    Sep 2007
    Location
    Essex
    Posts
    2,604
    Thank Post
    154
    Thanked 205 Times in 190 Posts
    Rep Power
    64
    Veeam does dedupe on it's backups (done at VM) level, wonder how that will work with a 2012 file server VM...

  11. #26

    SYNACK's Avatar
    Join Date
    Oct 2007
    Posts
    10,692
    Thank Post
    824
    Thanked 2,570 Times in 2,187 Posts
    Blog Entries
    9
    Rep Power
    731
    Dedupe just works by linking stuff that is the same to the same place in the behind the scenes bit of the file system. When it is read out it reads out as a stream including the 'multiple' files as it is transparent to the user or unaware client programs. It would then be deduped again by Veeam. No big mystery.

  12. #27

    Join Date
    Dec 2009
    Posts
    913
    Thank Post
    96
    Thanked 184 Times in 159 Posts
    Rep Power
    53
    I think we get around 35% space back with dedup on our NetApp

    We're a secondary school with 1200 pupils and around 150 staff, just a figure to work off for you.

  13. #28
    k-strider's Avatar
    Join Date
    Oct 2006
    Location
    Gloucester
    Posts
    357
    Thank Post
    7
    Thanked 40 Times in 30 Posts
    Rep Power
    23
    dedupe.jpg
    here is my 2012 server that hosts basically the student and staff shared area plus the network programs share. i've not got enough virtual servers on my 2012 server yet to get a good idea of the Dedupe rates there yet.

  14. #29

    Join Date
    May 2007
    Location
    Kent
    Posts
    121
    Thank Post
    35
    Thanked 10 Times in 9 Posts
    Rep Power
    27
    Very impressed with this feature. It is very easy to set up and has almost doubled the space available on my student drive. We have a large amount of data, and it was approaching 2TB. After running deduplication it is now around 1TB!

    dedupe.jpg

  15. Thanks to mitchell1981 from:

    gshaw (9th November 2012)

  16. #30
    gshaw's Avatar
    Join Date
    Sep 2007
    Location
    Essex
    Posts
    2,604
    Thank Post
    154
    Thanked 205 Times in 190 Posts
    Rep Power
    64
    Was impressed when this was on the feature list for 2012 and even more so from the feedback above, well done MS

SHARE:
+ Post New Thread
Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. De-fragmentation of the Exchange data store
    By nicholab in forum Enterprise Software
    Replies: 4
    Last Post: 4th July 2011, 03:53 PM
  2. Data retention
    By Dos_Box in forum School ICT Policies
    Replies: 21
    Last Post: 27th March 2006, 08:40 AM
  3. Cd Hidden Data
    By StewartKnight in forum Hardware
    Replies: 6
    Last Post: 7th November 2005, 10:45 AM
  4. Data Projector being stupid
    By e_g_r in forum Hardware
    Replies: 3
    Last Post: 19th October 2005, 03:27 PM
  5. Data Protection Act - re: Remote Access
    By mark in forum School ICT Policies
    Replies: 18
    Last Post: 26th September 2005, 07:19 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •