Personal Data Deduplication Software

[EduTech]

Hey Folks

 

As I am looking across hard drives that I have acquired over time where I have stored data but also backups of data I am trying to find a way where I can perform data depuplication across these drives so that I can look at the end delta and then look at backing this up to some cloud storage and remove the reliance upon TB of disk space which to be honest I very rarely access.

 

Do any of you have any recommendations on software that can do this for personal use (one time) as it is not something I have looked for before; I know plenty of enterprise software that can do this and I imagine none of our suppliers on here are willing to borrow me a license for a weekend :-)

 

Any Ideas?

 

Many Thanks,

James.

[Arthur]

One of the simplest ways this could be done is to create a VHDX, attach it to a Windows Server (2012 R2/2016), enable dedupe on that volume, copy all of your data over and then run the relevant dedupe PowerShell commands. Defrag and compact the VHDX, detach it and then upload to the cloud.

 

Johan Arwidmark used this technique to dedupe 183GB of VMs down to 16GB.

 

http://deploymentresearch.com/Research/Post/406/Beyond-Zip-How-to-store-183-GB-of-VMs-in-a-16-GB-file-using-PowerShell

 

If you want to dedupe at the file level, rather than block level you could use Duplicate File Hard Linker on the VHDX volume containing the files.

 

Another option would be to use a deduplicating archiver like zpaq. It can dedupe and compress the data so it should take up a lot less space than the method above.

 

When adding files, zpaq uses a rolling hash function to split files into fragments with an average size of 64 KB along content-dependent boundaries. Then it computes the SHA-1 hash of the fragment and compares it with saved hashes from the current and previous versions. If it finds a match then the fragment is not stored.

 

Deduplication requires 1 MB of memory per GB of deduplicated but uncompressed archive data to update, and 0.5 MB per GB to list or extract.

 

Note: Do not use the zpaq tool on disks that are already deduped with the Microsoft Data DeDuplication feature. It will cause corrupt archives.

Edited March 11, 2017 by Arthur