fiza Posted April 30, 2015 Posted April 30, 2015 I am setting up a new File server (2012 R2). I am looking at Data Deduplication which seems a no brainer. Silly questions : From what I can see it will take any duplicated files and remove all but one copy and replace other copies with pointers to the one copy left. Where is that copy kept? What happens if a user deletes the only copy left? Or does it not work like that?
localzuk Posted April 30, 2015 Posted April 30, 2015 The way it works is, the master data is moved to a single store (hidden) on the drive. The data is then pointed to by the files that the end user sees. I believe this is now block based, so it isn't files themselves, but blocks of data, so it is individual blocks of a file which are pointed to. If the final copy is deleted, the de-duplication scheduled task deletes any superfluous blocks of data when it is next ran. 1
Steve21 Posted April 30, 2015 Posted April 30, 2015 Effectively it makes a cache of "shared bits" that each file points to. So deleting one file doesn't affect the cache (Ignoring advanced bits of how it works with data scrubbing and garbage collection etc) Steve 1
fiza Posted April 30, 2015 Author Posted April 30, 2015 The way it works is, the master data is moved to a single store (hidden) on the drive. The data is then pointed to by the files that the end user sees. I believe this is now block based, so it isn't files themselves, but blocks of data, so it is individual blocks of a file which are pointed to. If the final copy is deleted, the de-duplication scheduled task deletes any superfluous blocks of data when it is next ran. So as long as one person has a copy of the file (or a pointer) then the "master" is safe? How does this affect backups? We currently use scripts to do our backups. Would we need to switch to some software package that understands deduplication?
localzuk Posted April 30, 2015 Posted April 30, 2015 So as long as one person has a copy of the file (or a pointer) then the "master" is safe? There isn't really a "master". There's a bunch of blocks of data and a file which links them all together. If there's nothing pointing at the blocks, they'll disappear when garbage collection happens. That said, in easy terms, yes that's right. How does this affect backups? We currently use scripts to do our backups. Would we need to switch to some software package that understands deduplication? It depends. If you use software that doesn't understand de-duplication then it will just copy the files as if they were full-fat files, creating a backup the full size of everything, with all files being real files. If you have backup software that understands the de-duplication, it will backup de-duplicated, making smaller and quicker backups. We currently use the prior - it doesn't really understand the de-duplication. When we next change our backup solution, we will be getting one that understands it so we can shorten backup times and have a smaller backup system.
Steve21 Posted April 30, 2015 Posted April 30, 2015 BackUp wise it depends where you store the backups too. As even after Veeam has done its magic, we then dedup the backup files as it's local on a 2012R2 server (not NAS etc). So even if your software doesnt recognise it, can still dedup the backups if local Steve
fiza Posted April 30, 2015 Author Posted April 30, 2015 BackUp wise it depends where you store the backups too. As even after Veeam has done its magic, we then dedup the backup files as it's local on a 2012R2 server (not NAS etc). So even if your software doesnt recognise it, can still dedup the backups if local Steve Our backups go onto a QNAP NAS.
Steve21 Posted April 30, 2015 Posted April 30, 2015 Our backups go onto a QNAP NAS. Is it Eth or iSCSI etc? As in if its presented to the server can still dedup on it Steve
fiza Posted April 30, 2015 Author Posted April 30, 2015 Is there any reason not to enable it? How does it deal with duplicate video files?
fiza Posted April 30, 2015 Author Posted April 30, 2015 Is it Eth or iSCSI etc? As in if its presented to the server can still dedup on it Steve Its iSCSI. Would it be better to Dedup the backups then?
Steve21 Posted April 30, 2015 Posted April 30, 2015 Generally it's only one certain things like SQL etc it says it shouldn't be enabled. We have dedup on all file servers and backups, and not noticed any downsides as long as you got the RAM etc to process it. BackUps currently running at 68% dedup'd (13.6TB volume, used 4.2TB so 9.2TB still free and have 9.9TB saved in dedup, so potentially another 18TB worth I can fit on ) Steve Steve
fiza Posted April 30, 2015 Author Posted April 30, 2015 Generally it's only one certain things like SQL etc it says it shouldn't be enabled. We have dedup on all file servers and backups, and not noticed any downsides as long as you got the RAM etc to process it. BackUps currently running at 68% dedup'd (13.6TB volume, used 4.2TB so 9.2TB still free and have 9.9TB saved in dedup, so potentially another 18TB worth I can fit on ) Steve Steve We have 16GB RAM on the server. enough?
Steve21 Posted April 30, 2015 Posted April 30, 2015 Yep should be fine (unless you have a massssive NAS ) we're running active at 11GB (32 total) for our 14TB volume. It's generally 1-2GB per TB (officially), but as it's only ever backups it's doing it runs pretty nice on the lower end. Steve
fiza Posted April 30, 2015 Author Posted April 30, 2015 Any recommendations as to backup software which supports deduplication?
Steve21 Posted April 30, 2015 Posted April 30, 2015 Depends what you mean by supports it, but Veeam if virtual servers There's dedupe-friendly compression in it so Veeam will compress it even further but not enough to effect dedupe. Steve
fiza Posted April 30, 2015 Author Posted April 30, 2015 Yep should be fine (unless you have a massssive NAS ) we're running active at 11GB (32 total) for our 14TB volume. It's generally 1-2GB per TB (officially), but as it's only ever backups it's doing it runs pretty nice on the lower end. Steve Our NAS is 40TB but only 10TB is presented to the File server for backup purposes. Depends what you mean by supports it, but Veeam if virtual servers There's dedupe-friendly compression in it so Veeam will compress it even further but not enough to effect dedupe. Steve Our file server is physical
clockend25 Posted April 30, 2015 Posted April 30, 2015 DPM technically supports de-dup, however there are known issues with the process which MS still hasn't sorted. It's incredibly unreliable for us.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now