ZFS - Deduplication (Dedup) Vs Compression

[dhicks]

I think your on the right path with your use of dedup

 

As in have a staging area that I copy files from over to the LessFS volume overnight? Now I've just got to find a bigger harddrive to make a useful sized staging area...

 

its just a pity you didn't have much success with ZFS.

 

I think ZFS worked fine, I think the problem was with Solaris on our backup server - I tried OpenSolaris before, a couple of years ago when I built the machine, and that time it didn't even boot from the install CD.

 

--

David Hicks

[dhicks]

have you tryed NexentaStor Project - CommunityEdition - NexentaStor Project

 

I remember trying something all-in-one and Solaris-based a few months ago, swearing at it lots, and using OpenSolaris instead. Could have been NexentaStor, I can't remember. I don't really care all that much which OS I wind up using, it's just until I get this server fixed I have no way to restore our media suite full of machines with mangled disk images...

 

--

David Hicks

[dhicks]

Right, I'm giving up on inline, block-level deduplicating file systems (ZFS, SDFS, LessFS, etc) and going back to a plain Ubuntu 9.10 server with an mdadm RAID-5 array containing an ext3 filesystem. Re-reading the documentation for rsync, it looks like it should be perfectly compatible with having a peridocally running script check for and eliminate duplicate files via means of hard links. Deduplication ratios on file system backups should be similar to what I'd have got with block-level deduplication anyway, and the 1:1.5 ratio I was getting on the 2GB disk-image files wasn't worth the massive performance hit - an order of magnitude for reads, and simply incredably slow for writes.

 

--

David Hicks