ZFS Compression Vs Deduplication (dedup)

Been playing with ZFS dedupe for the last two weeks and just wanted to share my findings.  

Setup OpenSolaris build 131
Sun X4200, 2 x Dual Core Opteron 2.6Ghz, 8Gb Ram, 4 x 73Gb SAS 10Krpm

root@osol:~# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
compress                    72K  66.9G    21K  /compress
dedupe                      72K  66.9G    21K  /dedupe

root@osol:~# zfs set compression=on compress
root@osol:~# zfs set dedup=on dedupe

Wanted to see how much real data would dedupe.

I loaded the my company  project/Software folders, 68,000 files (Visio/PDF/Project/Word/OpenOffice/Excel,ISO's... ) total of 38.9Gb

Load times, copying files from local UFS filesystem to ZFS dataset.

root@osol:/ufs# ptime tar cf - iso projects software | pv | ( cd /dedupe/ ; tar xf - )
real    19:51.930407394
user        5.807881662
sys      1:48.025965013
38.8GB 0:19:51 [33.3MB/s]

root@osol:/ufs# ptime tar cf - iso projects software | pv | ( cd /compress/ ; tar xf - )
real    18:46.544321180
user        3.368262960
sys      1:52.065809786
38.8GB 0:18:46 [35.3MB/s]

The deupe ZFS volume was 66 seconds slower than the compress volume.

Let see how much space we saved for both methods

root@osol:/ufs# zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
compress    68G  36.1G  31.9G    53%  1.00x  ONLINE  -
dedupe      68G  38.4G  29.6G    56%  1.02x  ONLINE  -
rpool       67G  49.8G  17.2G    74%  1.00x  ONLINE  -

root@osol:/ufs# zfs get compressratio compress
NAME      PROPERTY       VALUE  SOURCE
compress  compressratio  1.08x  -

The compressed volume did a better job than dedupe and saving an extra 6% storage.

Conclusion
There isn't any advantages for dedupe on a general home file share, slight slower performance and less space saved when compared to compression.

Now why would you want to dedupe ? Well just look at my dedupe ratio of 2.28 for a NFS share with VMware, now this is exciting!

root@osol:~$ zpool list vm-dedupe
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
vm-dedupe  68G   16.1G  51.9G    23%  2.28x  ONLINE  -

Therefore I can only say "Some data is more equal than others."

Andy
 (Minor edit  7.02.10)

Comments

Post a Comment

Popular posts from this blog

Solaris 11 Locale en_GB.UTF-8 / en_GB.ISO8859-1 / en_GB.ISO8859-15

When you install Solaris 11/11 from standard text installer cdrom, the en_GB-UTF-8 locales are not installed, even if you select British locales on install. EDIT 4th FEB 2018 - Still valid for Solaris 11.4 Beta.  To make sure packages are have the correct localisation, IPS uses facet as a localisation personality. $ pkg facet FACETS VALUE facet.locale.pt_BR True facet.locale.es_ES True facet.locale.en_US True facet.locale.zh_TW True facet.locale.zh_CN True facet.locale.de_DE True facet.locale.fr_FR True facet.locale.it_IT True facet.locale.ko_* True facet.locale.ja_* True facet.locale.es True facet.locale.de True facet.locale.zh True facet.locale.ko True facet.locale.it True facet.locale.pt True facet.locale.fr True facet.locale.en True facet.locale.ja True facet.locale.* False As you can see no en_GB is described and " facet.locale.* False " is a deny everything else rule. So let change this. I'm assuming you have a Solaris 11 repository already configured
Read more

Scheduled network capture on Windows using Wireshark (tshark.exe)

A customer had an iSCSI issue and was required to capture network packets at a specific time on a Windows 2008 server. I came up with simple method using Wireshark's tshark.exe and Windows scheduler "AT". at cmd /c c:\capture.bat type c:\capture.bat rem Capture WireShark example rem Andy Paton rem WTL rem use AT to run batch rem example at 01:50 cmd /c c:\capture.bat rem debug at issues rem example at 09:50 /interactive cmd /k c:\capture.bat rem -a duration:1200 in seconds rem -B Buffer Size - default is 1Mb rem -i Interface number - use "tshark.exe -D" to list interface numbers rem -n don't resolve IP addresses rem -q Quiet output rem -w output file rem capture filter "host " c:\"Program Files"\Wireshark\tshark -a duration:1200 -B 2 -i 4 -n -q -w c:\network.out host 192.168.1.1
Read more

[Linux] X-server ScreenShots from the CLI "ImageMagick"

Build some documentation for a customer and required some screenshots, KDE has Ksnaphot . But I wanted many screen shots and need a simple cli command to capture various screens. ImageMagik provides the " import " command which does just the job from the command line. To capture the whole desktop as a JPEG use the following. $ import -window root screen.jpg To save in a diffrent grapics format change the extention of the output file. (gif,pnm,pmm,tiff etc..) To capture single window ie firefox You need to find the window name, two methods of getting the "window name" 1. use xwininfo, which will then ask you to click target window and display information. 2. xlsclients -l display list of windows Note you can use a window title or the DECIMAL window ID, xwininfo and xlsclients output in HEX. $ import -window < WINDOW ID DECIMAL > screen1.jpg $ import -window "< WINDOWS TITLE >" screen2.jpg
Read more