ZFS - Deduplication (Dedup) Vs Compression

[apaton]

Some more basic tests on ZFS Dedup, this time to see what it like with real office data.

 

Setup OpenSolaris build 131

Sun X4200, 2 x Dual Core Opteron 2.6Ghz, 8Gb Ram, 4 x 73Gb SAS 10Krpm

 

Created two ZFS data sets, one with dedup enabled the other with compression. (default level)

[font=Courier New]root@osol:~# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
compress                    72K  66.9G    21K  /compress
dedupe                      72K  66.9G    21K  /dedupe[/font]
[font=Courier New]
root@osol:~# zfs set compression=on compress
root@osol:~# zfs set dedup=on dedupe[/font]  

Real data transfered to the drives.

I loaded the my company project/Software folders, 68,000 files (Visio/PDF/Project/Word/OpenOffice/Excel,ISO's... ) total of 38.9Gb

 

Load times, copying files from local UFS filesystem to ZFS dataset.

 

Copy data to a ZFS dedup dataset

[font=Courier New]root@osol:/ufs# ptime tar cf - iso projects software | pv | ( cd /dedupe/ ; tar xf - )
real    19:51.930407394
user        5.807881662
sys      1:48.025965013
38.8GB 0:19:51 [33.3MB/s] [/font]

Copy data to a ZFS compressed dataset

[font=Courier New]root@osol:/ufs# ptime tar cf - iso projects software | pv | ( cd /compress/ ; tar xf - )
real    18:46.544321180
user        3.368262960
sys      1:52.065809786
38.8GB 0:18:46 [35.3MB/s][/font]

The ZFS dedup dataset was 66 seconds slower than the compress volume for data transfer.

 

Let see how much space we saved for both methods

ZFS Dedup

[font=Courier New]root@osol:/ufs# zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
compress    68G  36.1G  31.9G    53%  1.00x  ONLINE  -
dedupe      68G  38.4G  29.6G    56%  [b]1.02x[/b]  ONLINE  -
[/font]

ZFS Compression

[font=Courier New] root@osol:/ufs# zfs get compressratio compress
NAME      PROPERTY       VALUE  SOURCE
compress  compressratio  [b]1.08x[/b]  -
[/font]  

Conclusion

The compressed dataset did a better job than dedup by a 6% saving in storage used. Therefore ZFS dedup doesn't deliver any real benefits for real "office data" and compression is a better offer. Also check this thread about compression on 7110.

 

Now why would you want to dedup? Well just look at my dedup ratio of 2.28 for a NFS share with VMware, now this is exciting!

 

[font=Courier New]root@osol:~$ zpool list vm-dedupe
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT 

vm-dedupe  68G   16.1G  51.9G    23%  [b]2.28x[/b]  ONLINE  -[/font] 

Therefore I can only say for deduplication "Some data is more equal than others"

 

It is possible to have dedup and compressions enabled on ZFS dataset, but I haven't tested that combination yet.

 

Andy

Ps Should Deduplication be shortened to "dedupe" or "dedup" ? I can't decided which.

Edited February 8, 2010 by apaton
Typo

[Soulfish]

I vote for dedupe

 

We're running compression on our 7110 and currently achieve a compression ratio of roughly 1.3x. We've got a mix of both LZJB and GZIP-2 depending on the share and the 7110 barley breaks a sweat

 

The data is a mix of VMs/Home Directories/General storage. I imagine the dedupe will give big benefits on things like shared storage and home directories where we've got people storing the same file across multiple locations.

[dhicks]

Now why would you want to dedup? Well just look at my dedup ratio of 2.28 for a NFS share with VMware, now this is exciting!

 

I'm going to try copying our workstation disk images (all of which are Windows XP, same software installed, just different workstations names) to our new OpenSolaris file server tomorrow, closely followed by the daily backups of our main file servers. I'll see how much of a space saving we get with that - with my file-level deduplicating backup system we've been able to store daily file server backups for the past two years, so hopefully we'll do better than that.

 

--

David Hicks

[Mindflux]

I'm going to try copying our workstation disk images (all of which are Windows XP, same software installed, just different workstations names) to our new OpenSolaris file server tomorrow, closely followed by the daily backups of our main file servers. I'll see how much of a space saving we get with that - with my file-level deduplicating backup system we've been able to store daily file server backups for the past two years, so hopefully we'll do better than that.

 

--

David Hicks

 

This is where dedupe is supposed to really help. Uncompressable data (videos, isos, etc) that have very little difference between them. This will get you considerably more space savings in this scenario over compression.

[apaton]

I'm going to try copying our workstation disk images (all of which are Windows XP, same software installed, just different workstations names) to our new OpenSolaris file server tomorrow, closely followed by the daily backups of our main file servers. I'll see how much of a space saving we get with that - with my file-level deduplicating backup system we've been able to store daily file server backups for the past two years, so hopefully we'll do better than that.

 

--

David Hicks

 

 

Just curious how you got on with this.

Did you get the space saving you expected?

 

Thanks

 

Andy

[dhicks]

Just curious how you got on with this. Did you get the space saving you expected?

 

Hmm. Since moving our machine images over to the ZFS filesystem I've kept getting image corruption issues. This could very well be to do with the client side of things rather than the server, but until I've figure out exactly what the problem is and how to stop it happening I'm going to be careful what I trust to the backup server for now.

 

--

David Hicks

[dhicks]

Hmm. Since moving our machine images over to the ZFS filesystem I've kept getting image corruption issues.

 

And double Hmm. After a reimage, non of the media suite machines want to boot. This isn't looking good...

 

--

David Hicks

[dhicks]

This isn't looking good...

 

...Does anyone want to recommend a deduplicating file system for Linux instead? I plan to move the backup server back to an Ubuntu Server install, but I see there's a couple of add-on filesystems available that do deduplication - ZFS for Linux, SDFS, maybe others that Google didn't find. Anyone any recommendations? SDFS looks pretty good, I'll probably try that first.

 

--

David Hicks

[pete]

And double Hmm. After a reimage, non of the media suite machines want to boot. This isn't looking good...

 

--

David Hicks

 

What file format are your images in? (I have a mental image of the RIS groveler service and ZFS clashing horribly).

Do you have before / after md5sums and what happens if you copy the image to a non-ZFS share?

[apaton]

Things still looking good this end. I'm running two VMware vSphere Windows XP VM on a dedup NFS mount.

Also checked some iso files with "digest -a md5 "

 

OpenSolaris Build 131.

 

Andy

[dhicks]

What file format are your images in?

 

Workstation images are created / restored from a Samba file share with Partimage, included on SystemRescueCD. Disk images written after the move to Solaris have a tendancy to be corrupted, while disk images created before that and simply copied over to the new server seem to be fine. The issue seems to be in writing data reliably to the Samba share on Solaris rather than the data storage itself - this could be something to do with disk I/O performance, or network I/O performance, or something else entirely.

 

--

David Hicks

[dhicks]

I'm copying my workstation images off the backup server on to an external harddrive in preparation for reinstalling with Ubuntu 9.10 Server. After removing some old directories of files, I notice that they are still being listed as present in the Samba share. The file system not being sure what has / hasn't been deleted would go quite a long way to explaining why disk images are getting currupted when they are being updated.

 

--

David Hicks

[pete]

There's always Debian-BSD? Debian GNU/kFreeBSD if you want ZFS with a debian userland and packages. Though if you're having issues working out whether it's samba or samba+solaris or samba+solaris+ZFS causing corruption, you may not want it.

 

Nexentastor bumped the limit of the community edition to 12TB recently as well. NexentaStor License Versions

[dhicks]

There's always Debian-BSD?

 

I know there's a ZFS-on-Linux project, but it doesn't seem to support deduplication yet. I'm just trying out a couple of FUSE-based file systems: SDFS (Opendedup), which doesn't seem to work on drives over 250GB, and LessFS, which looks a bit more promising. I'm now back to a Ubuntu 9.10 (64 bit version) and have an mdadm RAID-5 array of 6 500GB harddrives with an ext3 file system on which acts as the base storage for LessFS' FUSE-based file system. I'm just going to set up Samba using the LessFS file system and see what happens.

 

--

David Hicks

[dhicks]

I'm just going to set up Samba using the LessFS file system and see what happens.

 

Ah ha - turns out that if you're sharing any FUSE-based file system with Samba you need to remember to pass the "allow_other" option through to the FUSE file system when you mount it, otherwise Samba comes along, tries to mount the file share as the authenticated user and fails, giving a "path can not be found" message in Windows, which is very confusing. So, for instance, I have this line in /etc/rc.local to mount the LessFS file system:

 

/usr/local/bin/lessfs /etc/lessfs.cfg /data -o allow_other

 

You also need to edit /etc/fuse.conf and make sure the line "user_allow_other" is un-commented, which it isn't by default.

 

Samba is configured as detailed in this other post:

 

http://www.edugeek.net/forums/nix/48032-configuring-samba.html

 

The content of my /etc/lessfs.cfg file look like so:

 

BLOCKDATA_PATH=/mnt/md0/dta
BLOCKDATA_BS=1048576
#
BLOCKUSAGE_PATH=/mnt/md0/mta
BLOCKUSAGE_BS=1048576
#
DIRENT_PATH=/mnt/md0/mta
DIRENT_BS=1048576
#
FILEBLOCK_PATH=/mnt/md0/mta
FILEBLOCK_BS=1048576
#
META_PATH=/mnt/md0/mta
META_BS=1048576
#
HARDLINK_PATH=/mnt/md0/mta
HARDLINK_BS=1048576
#
SYMLINK_PATH=/mnt/md0/mta
SYMLINK_BS=1048576
#
LISTEN_IP=127.0.0.1
LISTEN_PORT=100
MAX_THREADS=2
# Cache size in megabytes.
CACHESIZE=128
# Flush data to disk after X seconds.
COMMIT_INTERVAL=30
#
MINSPACEFREE=10
# Consider SYNC_RELAX=1 or SYNC_RELAX=2 when exporting lessfs with NFS.
SYNC_RELAX=0
ENCRYPT_DATA=off
# ENCRYPT_META on or off, default is off
# Requires ENCRYPT_DATA=on and is otherwise ignored.
ENCRYPT_META=off

 

 

/mnt/md0 is simply a mount point for a mdadm array containing an ext3 filesystem, mounted via /etc/fstab:

 

#                                        
proc            /proc           proc            defaults                        0       0
/dev/sdg1       /               ext2            errors=remount-ro               0       1
/dev/sdg5       none            swap            sw                              0       0
/dev/scd0       /media/cdrom0   udf,iso9660     user,noauto,exec,utf8           0       0
/dev/fd0        /media/floppy0  auto            rw,user,noauto,exec,utf8        0       0
/dev/md0        /mnt/md0        ext3            defaults                        0       0

 

--

David Hicks

[pete]

Mind having a look to see how much performance overhead FUSE adds when you get a moment?

[dhicks]

Mind having a look to see how much performance overhead FUSE adds when you get a moment?

 

How would I tell?

 

I can tell you that delete performance is really slow - this is understandable, and pointed out in the documentation, but you tend to forget. I had to leave the server overnight to complete an "rm" operation on a 2GB image file. Not an issue on a backup server running batch jobs overnight, but not something for everyday use.

 

--

David Hicks

[dhicks]

I can tell you that delete performance is really slow

 

Copying files from a USB harddrive to the LessFS drive seems to be happening at around 6.5 Megabytes per second - so at this rate, my 54GB of disk image data is going to take around 6 days to copy. That's quite slow...

 

--

David Hicks

[dhicks]

Copying files from a USB harddrive to the LessFS drive seems to be happening at around 6.5 Megabytes per second

 

Ah, found the problem - default blocksize is 4K, which is great for deduplication but bad for disk performance. Setting block size to 128K has the file copy whizzing along - I'll maybe try 32K or 64K and see what the best size/performance tradeoff is.

 

--

David Hicks

[dhicks]

Setting block size to 128K has the file copy whizzing along

 

...but with a deduplication ratio of around 1:1.01. Switched to 64k blocks, current ratio around 1:10 - this is only from a quick script I wrote, I'm not sure if it's correct yet.

 

--

David Hicks

[dhicks]

Now on 32K block size, current deduplication ratio of 1:8 and my 54GB of disk image data should hopefully be copied over by midnight. I'm worried that write performance is going to go down the more data is stored, but I guess I'll find out (I'll come back tomorrow and check and see if the files have copied accross or not). LessFS is an inline deduplication system (SDFS had a batch-mode option, too), but a solution is to have data written to a non-lessFS volume and copy files over to the LessFS volume overnight, deduplicating as it goes. The documentation mentions having separate volumes to store block data and metadata (because this is a FUSE-based file system, so it stores its data as database files in an existing file system) - nice idea, but I don't have two separate RAID arrays to store data on (and storing metadata on a non-RAID disk seems a little pointless).

 

--

David Hicks

[apaton]

I've also come across the same block size issues with ZFS. (Sun S7000 refers blocks as database record). Smaller Blocks < 32K can give a better dedup ratio but requires more memory and Disk/Volume throughput is reduced for data transfers.

 

Personally I'm only 60% convinced that dedup for general users file systems is efficient use of resources on a NAS file severer.

 

General user filesystems at my customers sites all have different amount of users and data profiles, thus calculating dedup returns is almost impossible to predict. So its become let try and see! Not very scientific.

 

I do know dedup works best when you have a large amount of duplicated data such as backup and Virtual Machines images.

 

dhicks I think your on the right path with your use of dedup, its just a pity you didn't have much success with ZFS.

[Guest Guest]

dhicks - have you tryed NexentaStor Project - CommunityEdition - NexentaStor Project Seems pretty good to me.

[apaton]

I've known about Nexenta for a while, but must admit didn't know they had a Community Edition. Another one to have a look at but time is a finite resource.

[Guest Guest]

I've known about Nexenta for a while, but must admit didn't know they had a Community Edition. Another one to have a look at but time is a finite resource.

 

Ill save you some hassle then; it aint going to run properly on anything but decent server gear with 4gb+ ram. Tryed it on a cheapo Dell T105 and it was having none of it.