ZFS dedupe performance hit? (and implications for dedupe on 7110)

[pete]

Reading the opensolaris forums, I notice a few people are seeing significant performance hits when turning on dedupe. (OpenSolaris Forums : [zfs-discuss] ZFS Dedup Performance ... and Re: [zfs-discuss] Troubleshooting dedup performance).

 

The "fix" appears to be loads of ram or an SSD for cache, though it looks like it needs work ([zfs-discuss] $100 SSD = >5x faster dedupe).

 

I'm building out a test vm now to see if I get similar issues.

 

a) Has anyone else run into this?

b) Bearing in mind the 7110s have no SSD, will it be sensible to turn on dedupe?

[apaton]

Reading the opensolaris forums, I notice a few people are seeing significant performance hits when turning on dedupe. (OpenSolaris Forums : [zfs-discuss] ZFS Dedup Performance ... and Re: [zfs-discuss] Troubleshooting dedup performance).

 

The "fix" appears to be loads of ram or an SSD for cache, though it looks like it needs work ([zfs-discuss] $100 SSD = >5x faster dedupe).

 

I'm building out a test vm now to see if I get similar issues.

 

a) Has anyone else run into this?

b) Bearing in mind the 7110s have no SSD, will it be sensible to turn on dedupe?

 

I've played with De-dupe, but only under OpenSolaris build 128a running on VirtualBox, I haven't done any performance testing yet. I'm building X4200 with OpenSolaris b132 next week, I will let you know how I get on with performance.

 

With regards the OpenSolaris forum posts listed, I'm struggling to get my head around how SSD with L2ARC is improving performance with de-dupe. L2ARC is a read cache, I thought the performance hit would be from writing de-dupe data to disk and not reading de-duped data. So some homework for me to do.

 

With reference to 7110, I feel the limitation will be a combination of maximum 8Gb memory and no read SSD L2ARC. (L2ARC is a secondary read cache to main system memory)

 

But we are still in development and the code will change, its a wait and see.

 

Andy

[apaton]

Basic performance testing of dedupe

 

Dedup on a Sun X4200 Opteron 2 x Dual Core 2.6, 8Gb Memory & 4 x 73Gb SAS Drives (10Krpm)

 

OS

apaton@osol:/export/home/iso/linux$ uname -a
SunOS osol 5.11 snv_131 i86pc i386 i86pc Solaris

Created 2 ZFS pools (dedupe,zfs) and a UFS Filesystem

 

• /dedupe - ZFS with Dedup enabled (SHA256 - Verify off)
  
• /zfs - ZFS with Dedup disabled (Defaults)
  
• /ufs - UFS filesystem (Defaults)
  

 

Data files, (CentOS-4.4.ServerCD-x86_64 is duplicated 3 times) (7.5Gb)

 

apaton@osol:/export/home/iso/linux$ ls -ilh *
462 -rwxrwxrwx 1 root   root  601M 2010-02-01 16:31 CentOS-4.4.ServerCD-x86_64_A.iso
463 -rwxrwxrwx 1 root   root  601M 2010-02-01 16:48 CentOS-4.4.ServerCD-x86_64_B.iso
459 -rwxrwxrwx 1 root   root  601M 2010-02-01 13:28 CentOS-4.4.ServerCD-x86_64.iso
457 -rwxrwxrwx 1 root   root  3.6G 2010-02-01 13:27 CentOS-5.1-i386-bin-DVD.iso
455 -rwxrwxrwx 1 root   root  628M 2010-02-01 13:25 rhel-3-u6-i386-es-disc2.iso
464 -rwxr-xr-x 1 apaton staff  68M 2010-02-01 23:27 rhel-3-u7-i386-as-disc1.iso
461 -rwxrwxrwx 1 root   root   36M 2010-02-01 13:29 rhel-5.2-server-x86_64-dvd.iso
453 -rwxrwxrwx 1 root   root  177M 2010-02-01 13:22 RHEL4-U2-x86_64-ES-disc1.iso
458 -rwxrwxrwx 1 root   root  699M 2010-02-01 13:28 ubuntu-8.10-desktop-i386.iso
456 -rwxrwxrwx 1 root   root  638M 2010-02-01 13:25 ubuntu-8.10-server-i386.iso

Initial copy of files to DEDUP enable ZFS dataset.

apaton@osol:/export/home/iso/linux$ pfexec ptime tar cf - . | pv | ( cd /dedupe; tar xf - )
real     5:28.688240194
user        0.572326624
sys        14.953198730
[b]7.53GB 0:05:28 [23.5MB/s][/b]

Initial copy of files default ZFS dataset.

apaton@osol:/export/home/iso/linux$ pfexec ptime tar cf - . | pv | ( cd /zfs; tar xf - )
real     2:37.843718899
user        0.575755074
sys        14.397860363
[b]7.61GB 0:02:37 [49.3MB/s][/b]

Initial copy of files default UFS filesystem.

apaton@osol:/export/home/iso/linux$ pfexec ptime tar cf - . | pv | ( cd /ufs; tar xf - )
real     2:53.149996234
user        0.711298116
sys        14.034587988
[b]7.61GB 0:02:53 [  45MB/s][/b]

Third copy of files to DEDUP enable ZFS dataset.

apaton@osol:/export/home/iso/linux$ pfexec ptime tar cf - . | pv | ( cd /dedupe/d3; tar xf - )
real     2:49.402522619
user        0.859008763
sys        11.766251301
[b]7.53GB 0:02:49 [45.5MB/s]
[/b]
apaton@osol:/$ zpool list dedupe
NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
dedupe    68G  6.38G  61.6G     9%  [b]3.56x [/b] ONLINE  -

Summary

The above result look conclusive that a ZFS dataset with Dedup enabled will slow things down dramatically. But notice performance on dedpue did improve on second and third copies.

 

I've researched on the critical performance factors for dedup and memory seems critical. The deduplication table (DDT) needs to be held in memory(ARC) or at least SSD (L2ARC/Readzilla). Simple answer more hardware the better!

 

This is the same findings as Pete pointed out in his post. (I now understand them better )

 

So therefore in my opinion the 7110 will give usable but not great performance for dedup. Sun 7310 or above will be the best option for serious dedupe performance/requirements.

 

Andy

Edited February 2, 2010 by apaton
spelling mistake

[webman]

It would be interesting to see if there was a difference in the initial write speed for lots of smaller files (but similar large overall size) instead of ISOs. For example, lots of office documents, images and audio files to represent as close to a "real world" environment as possible.

[dhicks]

The "fix" appears to be loads of ram or an SSD for cache

 

I've just been investigating using OpenSolaris for our backup server. Our backup system currently handles de-duplication at the file level, calculating a hash value for each file, but block-level de-duplication would seem rather more space-efficient. Searching Google, I came accross this post:

 

ZFS Deduplication : Jeff Bonwick's Blog

 

Which points out that you can change the hashing algorithm used from SHA256 to, say, fletcher4, but make sure you turn on verification, i.e. make sure the file system checks for hash collisions. If memory or CPU performance is an issue you could pick a hashing algorithm that produced a range that would fit better in to RAM and/or took less time to calculate.

 

Going by the post above, block-level de-duplication has only been available in ZFS file systems since November just gone. The OpenSolaris install available on Sun's site seems to be from back in June. Can I actually use de-duplication right now on OpenSolaris, does it install straight from the CD, or do I have to do something complicated to install de-duplication support?

 

--

David Hicks

[dhicks]

you can change the hashing algorithm used from SHA256 to, say, fletcher4, but make sure you turn on verification

 

Okay, scratch that, just read the FAQ - seems like that's not actually working yet.

 

--

David Hicks

[pete]

Can I actually use de-duplication right now on OpenSolaris, does it install straight from the CD, or do I have to do something complicated to install de-duplication support?

 

--

David Hicks

 

Look here for newer builds: Genunix

 

I also found the EON nas build, but only really poked around with it to see if it worked. E O N

 

I suppose we could always campaign for Sun to let us bump up the ram on the 7110s. It's not like there aren't free slots in the chassis.

 

On ZFS I was hoping for a delayed dedupe functionality (as you can have with file-based dedupe), so it could spend the weekend/overnight sorting out duplicates.

[dhicks]

Can I actually use de-duplication right now on OpenSolaris, does it install straight from the CD, or do I have to do something complicated to install de-duplication support

 

Okay, so from reading the de-duplication FAQ, it seems I need "SXCE build 128 or 129". Does anyone know what SXCE is or how I go about installing it on a PC or virtual machine?

 

--

David Hicks

[Soulfish]

Look here for newer builds: Genunix

 

I also found the EON nas build, but only really poked around with it to see if it worked. E O N

 

I suppose we could always campaign for Sun to let us bump up the ram on the 7110s. It's not like there aren't free slots in the chassis.

 

On ZFS I was hoping for a delayed dedupe functionality (as you can have with file-based dedupe), so it could spend the weekend/overnight sorting out duplicates.

 

I'd agree that it'd be a real shame if we couldn't use Dedupe on the 7110 series. Here's hoping that we're allowed to bump the ram up!

[CyberNerd]

is anyone using dedupelication ZFS with samba homedirectories? We been using samba for some time now, and I'd certainly consider a solaris migration just wondered if anyone had success here

[apaton]

Okay, so from reading the de-duplication FAQ, it seems I need "SXCE build 128 or 129". Does anyone know what SXCE is or how I go about installing it on a PC or virtual machine?

 

--

David Hicks

 

 

Build 128 was first to have Dedupe, 128a was released to fix bug in Fletcher4.

Current binary release is build 131.

 

Upgrade to latest build release.

 

pfexec pkg set-authority -O http://pkg.opensolaris.org/dev/ opensolaris.org

pfexec pkg image-update

 

Andy

[dhicks]

you can change the hashing algorithm used from SHA256 to, say, fletcher4, but make sure you turn on verification

 

Okay, scratch that, just read the FAQ - seems like that's not actually working yet.

 

Build 128 was first to have Dedupe, 128a was released to fix bug in Fletcher4.

 

Okay, un-scratch the above, then, it seems it does work. If / when I get OpenSolaris working I guess I'll find out myself - no luck so far, EON booted just fine as a live CD but seems to be missing instructions on how to get it on to my harddrive, the OpenSolaris CD didn't get past GRUB, and I managed to screw up the CentOS install to run OpenSolaris as a Xen VM 3 times in one day. I'll try again tomorrow.

 

--

David Hicks

[apaton]

... and I managed to screw up the CentOS install to run OpenSolaris as a Xen VM 3 times in one day. I'll try again tomorrow.

 

Give VirtualBox a try, worked first time for me.

[pete]

Okay, un-scratch the above, then, it seems it does work. If / when I get OpenSolaris working I guess I'll find out myself - no luck so far, EON booted just fine as a live CD but seems to be missing instructions on how to get it on to my harddrive,

 

--

David Hicks

 

Start here: EON ZFS Storage (NAS) (EON ZFS Storage) halfway down the page. Specifically:

After the image (eon.iso) is burned to a CD and booted. Login info is:

user: admin pass: eonstore

user: root pass: eonsolaris

 

Type and run the following. This script prompts the user through configuration questions like hostname, IP/DHCP, netmask, domain name and more. This step will ask questions to configure and ID the system for live image use.

# /usr/bin/setup

 

This step is optional but necessary if the configuration changes made are to be preserved beyond a reboot or power off. This requires a writable destination, USB or CF drive attached before the command is run. The command will facilitate formatting and installing the live image (image on the CD) to the USB or CF drive.

# /usr/bin/install.sh

 

This step should be done after install.sh or, to preserve configuration changes made to the image. This preserves the original image to /mnt/eonX/boot/x86.eon.orig (bootable by the OEM choice from GRUB) and saves a new default boot image to /boot/x86.eon. It will move the live image to x86.eon.1, x86.eon.2 and so on each time it is run.

# /usr/bin/updimg.sh

[Guest Guest]

I also found the EON nas build, but only really poked around with it to see if it worked. E O N

 

 

Ive found EON to be by far the easiest and most reliable for doing testing in virtualbox.

Theres a web interface for it too. No substitute for CLI quite yet but great for doing general mantainance on an OS you most likely never hardly touch

 

// napp-it free ZFS NAS-SAN-Server: installed quickly - ready to run - easy to manage

[dhicks]

Give VirtualBox a try, worked first time for me.

 

Nope, no luck here. OpenSolaris refuses to boot any further than a GRUB command prompt. I've tried running it on the bare server, as an HVM Xen VM and as a VirtualBox VM on my desktop - same results every time. Hmm, maybe a corrupt ISO? I'm currently donwloading osol-dev-131-x86.iso from Genunix sintead, see if I get anywhere with that.

 

--

David Hicks

[dhicks]

/usr/bin/updimg.sh

 

This script fails to run on my backup server. In the end, I figured out this was due to a lack of RAM. The LiveCD boots and assigns /tmp to a RAM disk. The script above then tries to decompress the filesystem on the CD to /tmp, so if you don't have enough RAM it fills up and breaks. A VirtualBox VM with 1.5GB of RAM assgined seems to work okay (the backup server has 1GB of RAM), so it looks like around 1.5GB is a sensible minimum to use this script.

 

EON is a good idea, but I don't think it's quite suitible for what I want to do - I don't really want to have to rebuild the boot file system every time I make a settings change, and I want to be able to setup Samba and install rsync without to much effort.

 

--

David Hicks

Edited February 5, 2010 by dhicks

[Guest Guest]

Nope, no luck here. OpenSolaris refuses to boot any further than a GRUB command prompt. I've tried running it on the bare server, as an HVM Xen VM and as a VirtualBox VM on my desktop - same results every time. Hmm, maybe a corrupt ISO? I'm currently donwloading osol-dev-131-x86.iso from Genunix sintead, see if I get anywhere with that.

 

--

David Hicks

 

Ive found the 64bit versions wont boot, infact ive even had it crash my host pc! 32bit has worked just fine.

[dhicks]

Ive found the 64bit versions wont boot

 

I checked the MD5 checksum, turned out to be a corrupt ISO image. I've now downloaded and installed 2009.06 (the latest release version booted but crashed on GUI login), updated it to the current dev version (er, I think - to a version that supports ZFS with deduplication support, anyway, which is what this is all in aid of in the first place) and managed to create a 6-disk RAID-Z array with dedupe switched on. Now I just have to get Samba working...

 

--

David Hicks

[apaton]

I think - to a version that supports ZFS with deduplication supports

 

Check your ZFS Version

apaton@osol:~$ zpool upgrade -v
This system is currently running ZFS pool version 22.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
1   Initial ZFS version
2   Ditto blocks (replicated metadata)
3   Hot spares and double parity RAID-Z
4   zpool history
5   Compression using the gzip algorithm
6   bootfs pool property
7   Separate intent log devices
8   Delegated administration
9   refquota and refreservation properties
10  Cache devices
11  Improved scrub performance
12  Snapshot properties
13  snapused property
14  passthrough-x aclinherit
15  user/group space accounting
16  stmf property support
17  Triple-parity RAID-Z
18  Snapshot user holds
19  Log device removal
20  Compression using zle (zero-length encoding)
21  Deduplication
22  Received properties

For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.

[dhicks]

Check your ZFS Version

 

Done that - setting dedupe failed with a "must upgrade" message, so I upgraded and all seems fine.

 

--

David Hicks