+ Post New Thread
Page 3 of 3 FirstFirst 123
Results 31 to 40 of 40
General Chat Thread, Deduplication - anyone else using it? in General; I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful. ...
  1. #31

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,803
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224
    I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful. You need to look at your workloads and check if dedupe is really worth it to you. In my case the system overhead caused by running deduping was not worth the performance hit. So when hard drives are so cheap per GB in the long run I found it better overall not to dedupe and just throw more disk space at the problem.

  2. #32

    john's Avatar
    Join Date
    Sep 2005
    Location
    London
    Posts
    10,530
    Thank Post
    1,496
    Thanked 1,050 Times in 919 Posts
    Rep Power
    303
    Quote Originally Posted by Geoff View Post
    I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful. You need to look at your workloads and check if dedupe is really worth it to you. In my case the system overhead caused by running deduping was not worth the performance hit. So when hard drives are so cheap per GB in the long run I found it better overall not to dedupe and just throw more disk space at the problem.
    Would agree with Geoff on this the overhead for de-dupe on the Sun / Oracle SANs were not worth it for us, although the space saved would be nice the loss of performance was not acceptable so it was turned back off.

  3. #33

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,803
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224
    I only really use it on our (120Tb) archive system currently. It is rarely accessed and using space efficiently is more important than speed of access. This system also runs with compression on some parts of the file system although this is rapidly becoming an irrelevance as file formats start to adopt compression natively (wav -> mp3 and office 97 -> office XML). Our day to day working file set (6Tb) runs uncompressed and with no deduplication.

  4. #34


    Join Date
    Feb 2007
    Location
    51.403651, -0.515458
    Posts
    8,868
    Thank Post
    226
    Thanked 2,666 Times in 1,965 Posts
    Rep Power
    785
    Quote Originally Posted by Geoff View Post
    I've run Dedupe on ZFS on a BSD Fileserver before now and I have to warn you to be careful.
    Dedupe on Server 2012 works in a totally different way compared to ZFS. The performance hit is extremely minimal* and you don't need huge amounts of RAM. Read the TechNet links below for details.


    * Assuming you don't do something stupid like setting MinimumFileAgeDays to 0 which forces Windows to constantly dedupe the volume(s) that have it enabled.

    For volumes with a lot of redundant data, the space savings can be massive.



    @Geoff. I'm guessing your Oracle SAN was under-specced for the amount of data you were deduping? From what I have read, 1TB of deduplicated data (4KB average block size) would require around 80GB RAM to hold the dedupe tables and need a beast of a CPU to do the SHA256 checksum calculations.
    Last edited by Arthur; 31st May 2013 at 11:50 AM.

  5. #35

    FN-GM's Avatar
    Join Date
    Jun 2007
    Location
    UK
    Posts
    15,938
    Thank Post
    886
    Thanked 1,693 Times in 1,472 Posts
    Blog Entries
    12
    Rep Power
    447
    Got it in production here.

    33% on file server
    55% on VLE server.

    No performance hits either.

  6. #36

    Join Date
    Jul 2006
    Location
    London
    Posts
    1,259
    Thank Post
    111
    Thanked 242 Times in 193 Posts
    Blog Entries
    1
    Rep Power
    74
    Long long ago I co-hosted my user data with the Windows 2000 RIS service.

    This meant i got de-dupe on my user shares. It wasn't entirely painless, and about 6 months after I first encountered a non-critical problem Microsoft added a note to say that it was not a recommended configuration. I never lost data, but restores could be 'fun'. It gave me about 40% more space that I had paid for and kept backup sizes down.

    I'm intrigued to see it come back into the Windows File Server role 14 years after it's initial outing. Looks like they've done a lot of work on it. Data Deduplication Overview

    In this incarnation I'd be surprised if it isn't a by-product of work done making Azure VMs space efficient.
    Last edited by psydii; 31st May 2013 at 12:04 PM.

  7. #37

    Geoff's Avatar
    Join Date
    Jun 2005
    Location
    Fylde, Lancs, UK.
    Posts
    11,803
    Thank Post
    110
    Thanked 583 Times in 504 Posts
    Blog Entries
    1
    Rep Power
    224
    Quote Originally Posted by Arthur View Post
    @Geoff. I'm guessing your Oracle SAN was under-specced for the amount of data you were deduping? From what I have read, 1TB of deduplicated data (4KB average block size) would require around 80GB RAM to hold the dedupe tables and need a beast of a CPU to do the SHA256 checksum calculations.
    It's not an Oracle SAN. However the front end server has 128Gb of RAM and Intel Xeons with 4x4 Cores. It also has a hardware card to offload the SHA256. So it's not like I didn't try. However 2012's dedupe runs on a schedule does it not? Because we're a 24/7 operation so I would need it to run real-time. Regardless, I just gave up on the idea and threw more storage at the problem.

  8. #38
    ADMaster's Avatar
    Join Date
    May 2012
    Posts
    327
    Thank Post
    5
    Thanked 33 Times in 28 Posts
    Rep Power
    23
    I have deduplication enabled on the data drive for my win8 box at home. I see between 55 – 60 % savings.

    I will be getting Server 2012 later this summer and plan to move all my shares to it. The eval tool says I’ll get 58% on my WDS / Software share and 27% on my user data share.
    @zbjsy
    From an elevated PowerShell session these commands will get you the info you’re looking for.

    Get-DedupStatus
    Get-DedupVolume

    Deduplication Cmdlets in Windows PowerShell

  9. #39

    FN-GM's Avatar
    Join Date
    Jun 2007
    Location
    UK
    Posts
    15,938
    Thank Post
    886
    Thanked 1,693 Times in 1,472 Posts
    Blog Entries
    12
    Rep Power
    447
    Quote Originally Posted by Geoff View Post
    However 2012's dedupe runs on a schedule does it not?
    It can do and/or it can run in the background. When the server is under load it will pause dedupe automatically.
    Last edited by FN-GM; 31st May 2013 at 02:39 PM.

  10. #40

    localzuk's Avatar
    Join Date
    Dec 2006
    Location
    Minehead
    Posts
    17,680
    Thank Post
    516
    Thanked 2,451 Times in 1,897 Posts
    Blog Entries
    24
    Rep Power
    832
    Just moved our staff files over to our 2012 server, and deduplication has run on the data. Total saving of 47% or 478GB.

SHARE:
+ Post New Thread
Page 3 of 3 FirstFirst 123

Similar Threads

  1. Lanschool - Anyone that uses it please help
    By mattmoo in forum Network and Classroom Management
    Replies: 16
    Last Post: 7th December 2011, 11:43 AM
  2. Anyone here used IT's Learning as their VLE?
    By Sam_Brown in forum Virtual Learning Platforms
    Replies: 9
    Last Post: 13th September 2010, 09:33 PM
  3. Avatar 3D - Anyone else seen it?
    By flyinghaggis in forum General Chat
    Replies: 44
    Last Post: 1st February 2010, 01:43 PM
  4. Replies: 8
    Last Post: 10th January 2008, 09:34 AM
  5. Replies: 1
    Last Post: 11th May 2007, 09:18 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •