Frazzle Posted May 14, 2018 Posted May 14, 2018 We have around 8TB of data and using Treesize I can see quite a bit of duplication. Can someone explain Deduplication in simple terms please? Is it to reduce the size of backups or to recoup Storage space on your File Server?
Jawloms Posted May 14, 2018 Posted May 14, 2018 We have around 8TB of data and using Treesize I can see quite a bit of duplication. Can someone explain Deduplication in simple terms please? Is it to reduce the size of backups or to recoup Storage space on your File Server? Yes to both. The way I understand it, if you have a file called "My really important data.doc" 17 times through the same drive, it lilterally is 17 times throughout that drive. Deduplication removes 16 of them, and then makes the pointer for those 16 point at the 17th, so you only have one of them taking up space.
JackCPickup Posted May 14, 2018 Posted May 14, 2018 It also counts for OS files in your VMs so it will dedup that storage massively. Some hyperconvergence(dedup appliance built in) and dedup appliances boast 60-90% dedup rate!
HPlum78 Posted May 14, 2018 Posted May 14, 2018 (edited) In a little more depth - The data is broken into chunks each chunk is looked at to identify duplicate chunks and then only one chunk is maintained and compressed. Any redundant chunks are replaced with a reference to a single copy of the chunk and each file is replaced with a reparse point that references its data chunks. You essentially end up with a chunk store, optimized files, un-optimized files and the free space due to the chuck data size and the compression used. If a user edits/ makes changes their copy of the file its only the delta that is stored not a complete copy of the original file. This will save you space your backup solution will need to be able to backup a deduplicated volume mind. In fact here is some useful info - https://msdn.microsoft.com/en-us/library/hh769303(v=vs.85).aspx and even more detail here https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand Edited May 14, 2018 by HPlum78
Frazzle Posted May 14, 2018 Author Posted May 14, 2018 Yes to both. The way I understand it, if you have a file called "My really important data.doc" 17 times through the same drive, it lilterally is 17 times throughout that drive. Deduplication removes 16 of them, and then makes the pointer for those 16 point at the 17th, so you only have one of them taking up space. Excuse my ignorance but if 16 pointers point to the 17th version of the file then is it possible for someone to delete the 17th and then all the pointers are useless? In very simple terms UserA has the file UserB has the file With Dedup both files point to 1 copy UserA decides he doesnt need the file anymore and deletes it. Is it still there for UserB?
Jawloms Posted May 14, 2018 Posted May 14, 2018 Excuse my ignorance but if 16 pointers point to the 17th version of the file then is it possible for someone to delete the 17th and then all the pointers are useless? In very simple terms UserA has the file UserB has the file With Dedup both files point to 1 copy UserA decides he doesnt need the file anymore and deletes it. Is it still there for UserB? I'll try and clarify. There is one file belonging to no-one, and 17 pointers pointing to it. If someone deletes one then they are simply deleting the pointer. When the last gets deleted, the file goes too. If what you've suggested could happen, there'd be no point would there. They did think of that!
jtotheb Posted May 14, 2018 Posted May 14, 2018 With Dedup both files point to 1 copy UserA decides he doesnt need the file anymore and deletes it. Is it still there for UserB? Yes. The data is still referenced so it isn't deleted.
colacao82 Posted May 14, 2018 Posted May 14, 2018 Are there any files types you should exclude from dedupe? i know about SQL and Exchange databases but any file like .zip etc?
HPlum78 Posted May 14, 2018 Posted May 14, 2018 (edited) Don't think of files as files they are chunks of data, the file that the users sees is just a list of pointers to those chunks. Look at the second link it gives an animated view of what happens when files are on a deduped volume. Edited May 14, 2018 by HPlum78
HPlum78 Posted May 14, 2018 Posted May 14, 2018 (edited) Movie files don't dedupe very well you can run the following Measure-DedupFileMetadata this will give you an indication of what you can get back by enabling dedupe on a volume. oh and encrypted files... Edited May 14, 2018 by HPlum78 1
PotNoodleTech Posted June 4, 2018 Posted June 4, 2018 So who is running Dedupe in anger on their network? Are all the data corruption bugs ironed out now? Want to try it on my backup servers but am a bit "mmmm" as I don't want any chance of corruption!!
maark Posted June 4, 2018 Posted June 4, 2018 I am running it here and at previous college on staff/student folders/shared drives - never had any issues. 1
Norphy Posted June 4, 2018 Posted June 4, 2018 I've been running it for a very long time. The only gotcha I've come across is when the volume is almost full. When a file is removed, the space that the file takes up isn't immediately returned to the file system. The dedupe service has to run a garbage collection periodically to reclaim this space. When there's lots of space available on the volume, this isn't an issue but when the volume is almost full, you have to be more aggressive with your garbage collection schedules. 1
andy_b Posted June 4, 2018 Posted June 4, 2018 I've been running it for a very long time. The only gotcha I've come across is when the volume is almost full. When a file is removed, the space that the file takes up isn't immediately returned to the file system. The dedupe service has to run a garbage collection periodically to reclaim this space. When there's lots of space available on the volume, this isn't an issue but when the volume is almost full, you have to be more aggressive with your garbage collection schedules. We sometimes just force an update with powershell if we have this problem: https://kallesplayground.wordpress.com/useful-stuff/windows-server-2012-r2-data-deduplication/
PotNoodleTech Posted June 5, 2018 Posted June 5, 2018 Cool - one more question are any of you running Dedupe as well as Bitlockering the drive? Wondering about possible performance issues? Cheers!
brougham Posted June 5, 2018 Posted June 5, 2018 We have and not noticed any issues. Bitlocker is at the time of write but dedupe (on Server 2012R2 at least) lags behind. It runs a task when the server isn't busy to dedupe. Described here- https://redmondmag.com/articles/2014/03/13/data-deduplication-in-windows-server.aspx Setup and a few settings to consider here- https://blogs.technet.microsoft.com/canitpro/2013/04/29/step-by-step-enabling-data-deduplication-on-windows-server-2012-volumes/ 1
Oaktech Posted June 5, 2018 Posted June 5, 2018 I just turned it on on my file servers and my backup server - saves me between 30 and 60% and not noticed any issues at all. The backup drive is bitlockered as well because it's a physical disk that could be walked off with. The file server drives are virtual in a SAN and the SAN firmware deals with drive encryption for me. 1
PotNoodleTech Posted June 6, 2018 Posted June 6, 2018 Well I am enabling it on my backup server which currently has two volumes, D with backups on it, and E empty ready for backup copies. I thought I would run the Dedupe Eval tool on both volumes to double check as it was recommended in the guide I am following. On the blank drive it runs fine and says drive is blank. on the used drive it starts running (drive lights churning) and then after 5 mins, crashes with an error. Scared to actually enable dedupe on this volume now incase it b0rks. Anyone come across this?
colacao82 Posted July 31, 2018 Posted July 31, 2018 I'm about to enable dedupe on my file server. Apart from compressed files, SQL and Exchange DB files. What other file types would you recommend excluding? I found one list but it was pretty much everything, which seems to defeat the purpose. It included pretty much all MS Office extensions... aac,aif,aiff,asf,asx,au,avi,flac,m3u,mid,midi,mov,mp1,mp2,mp3,mp4,mpa,mpe,mpeg,mpeg2,mpeg3,mpg,ogg,qt,qtw,ram,rm,rmi,rmvb,snd,swf,vob,wav,wax,wma,wmv,wvxaccdb,accde,accdr,accdt,docm,docx,dotm,dotx,pptm,potm,potx,ppam,ppsx,pptx,sldx,sldm,thmx,xlsx,xlsm,xltx,xltm,xlsb,xlam,xllace,arc,arj,bhx,bz2,cab,gz,gzip,hpk,hqx,jar,lha,lzh,lzx,pak,pit,rar,sea,sit,sqz,tgz,uu,uue,z,zip,zoo,edb,jrs Can I include Office files in my dedupe? Thanks
Arthur Posted July 31, 2018 Posted July 31, 2018 Can I include Office files in my dedupe? Office documents are fine. Microsoft even shows them in their dedupe documentation. https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now