deduplication of mapped drives

[CyberNerd]

Is it worth it?

 

I've done some experiments, similar to this thread of 2 yrs ago:

http://www.edugeek.net/forums/nix/49844-zfs-deduplication-dedup-vs-compression.html

 

using zfs block level deduplication. Just using the shared drives I had a dedup ration of 1.16% (around a terabyte of data in total). If I added in staff homedrives to the pool the ratio went upto about 1.80%

 

I can see some advantages:

 

nearly doubles the amount of disk space available with compression

deduplication over the wire means really fast backups

simple to add new disks to a zfs pool and increase size

self healing filesystem

handy for shared files if google drive== homedrive

 

and some disadvantages, mostly learning related:

 

learning freebsd/solaris over linux

different quota system

not getting full benefit from hardware raid5 on SAN (I presented the disks separately to the server so that I'm not causing performance issues by raiding it twice)

performance/RAM overhead

disks are cheap anyway, saving 1TB only really saves me a couple of hundred quid.

 

I keep going around in circles with it. what should I do?

Edited May 1, 2012 by CyberNerd

[dhicks]

Just using the shared drives I had a dedup ration of 1.16% (around a terabyte of data in total). If I added in staff homedrives to the pool the ratio went upto about 1.80%

 

It'd be interesting to see how that changed for media - photos, video and audio. People have a tendancy to dump stuff from camera memory cards onto shared file areas any old how, and having a deduplicated file system would avoid that becoming too much of a problem.

 

We've just set up a new file server using FreeNAS, which is based on FreeBSD and uses ZFS. We don't have enough RAM in the machine to use deduplication, though. The management interface to set up file shares and so on seems to be quite usable - we seem to have file areas with appropriate Windows file permissions and so on. I seem to remember snapshot performance under ZFS being considered rather better than with LVM under Linux - LVM's write performance drops with each snapshot you do. You can still SSH in to the machine to run scripts and so on, there shouldn't be that much actual BSD stuff to deal with.

 

not getting full benefit from hardware raid5 on SAN (I presented the disks separately to the server so that I'm not causing performance issues by raiding it twice)

 

If you have the time to compare, you could try ZFS on a hardware RAID array and tell us the results. Would a JBOD array give you the write perfomance of the RAID array without the RAID-5 overhead/complications?

[Arthur]

The following article might help...?

 

http://nex7.com/readme1st