Behind the scenes we use ZFS as storage for our offsite backups.
\nWe have backups in 2 separate physical locations + original data on the server(s), as data is mui importante!
\nZFS is a rather nice storage file system that improves radically on older RAID based solutions, offering a lot more funky options, like snapshot’s (where the OS can store have multiple versions of files, similar to Time Machine backups), and more importantly compression.<\/p>\n
At some point we’ll be deploying a SAN (storage area network) on a blade server in the data center for data using ZFS, using lots of three and four letter acronyms –
\nESXi for base OS, then a VM running providing ZFS storage and iSCSI targets with hardware passthru for other VM’s, then other blades in the server doing clustering.
\nRight now we’re waiting on an LSI SAS card (see more anacronyms!), so we can deploy…, but I digress.<\/p>\n
Back to ZFS.<\/p>\n
The ZFS version we use now allows flags, yay!, and that means we can choose alternate compression methods.
\nThere is a reasonably newish compression algorithm called LZ4 that is now supported, and it improves both read and write speeds over normal uncompressed ZFS, and some benefits over compressed ZFS using the standard compression algorithm(s).<\/p>\n
To quote: “LZ4 is a new high-speed BSD-licensed compression algorithm written by Yann Collet that delivers very high compression and decompression performance compared to lzjb (>50% faster on compression, >80% faster on decompression and around 3x faster on compression of incompressible data)”<\/p>\n
First up check if your zfs version supports it:<\/p>\n
The following features are supported:<\/p>\n FEAT DESCRIPTION Mine does (well duh!)<\/p>\n So…, I can turn on support.<\/p>\n As a note, lz4 is not backward compatible, so you will need to use a ZFS version that supports flags *and* lz4. If you aren’t sure, check first with the command above and see if there is support. My storage pools are typically called nas or tank I have nas and nas\/storage so I did - <\/p>\n zpool set feature@lz4_compress=enabled nas Once the flag is set though, you can set compression on the pools or volume. If you set at the storage volume level, then new pools inherit the compression setting.<\/p>\n Here are my volumes \/ pools<\/p>\n I’ve already set compression on (although it doesn’t take effect till I copy new data onto the pools \/ volumes). If I create a new pool you’ll see it gets created with the same compression inherited from its parent storage volume.<\/p>\n I’ll copy some dummy data onto there, then recheck.<\/p>\n Nice!<\/p>\n Obviously, compression ratio’s will depend highly on the data, but for our purposes, most things are web data, mail and other things, so we’re heavy on text content, and benefit highly from compression.<\/p>\n Once we get our SAN up and running, I’ll be looking at whether I should be using rsync still or I should look at zfs snapshots -> zfs storage on other servers. <\/p>\n","protected":false},"excerpt":{"rendered":" Behind the scenes we use ZFS as storage for our offsite backups. We have backups in 2 separate physical locations + original data on the server(s), as data is mui importante! ZFS is a rather nice storage file system that improves radically on older RAID based solutions, offering a lot more funky options, like snapshot’s […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[25,4],"tags":[208,482,481,441],"class_list":["post-1117","post","type-post","status-publish","format-standard","hentry","category-technical-mumbo-jumbo","category-useful-info","tag-backups","tag-compression","tag-lz4","tag-zfs"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/1117","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/comments?post=1117"}],"version-history":[{"count":1,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/1117\/revisions"}],"predecessor-version":[{"id":1118,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/posts\/1117\/revisions\/1118"}],"wp:attachment":[{"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/media?parent=1117"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/categories?post=1117"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.computersolutions.cn\/blog\/wp-json\/wp\/v2\/tags?post=1117"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nzpool upgrade -v
\nThis system supports ZFS pool feature flags.<\/p>\n
\n-------------------------------------------------------------
\nasync_destroy (read-only compatible)
\n Destroy filesystems asynchronously.
\nempty_bpobj (read-only compatible)
\n Snapshots use less space.
\nlz4_compress
\n LZ4 compression algorithm support.
\n<\/code><\/p>\n
\nAt the time of writing nas4free doesn’t support it, zfsonlinux does though, as does omnios, illumos and other solaris based OS’s.<\/p>\n
\nNext step is to turn on that feature.<\/p>\n
\nTo enable lz4 compression, its a 2 step process.<\/p>\nzpool set feature@lz4_compress=enabled
\nzpool set compression=lz4
\nzpool set compression=lz4 nas
\nzpool set compression=lz4 nas\/storage
\n<\/code><\/p>\nzfs list
\nNAME USED AVAIL REFER MOUNTPOINT
\nnas 6.09T 4.28T 209K \/nas
\nnas\/storage 6.09T 4.28T 6.09T \/nas\/storage<\/code><\/p>\n
\nWe can check compression status by doing a zfs get all command, and filtering by compress<\/p>\nzfs get all | grep compress
\nnas compressratio 1.00x -
\nnas compression lz4 local
\nnas refcompressratio 1.00x -
\nnas\/storage compressratio 1.00x -
\nnas\/storage compression lz4 local
\nnas\/storage refcompressratio 1.00x -
\n<\/code><\/p>\nzfs create nas\/test
\nroot@nas:\/nas# zfs get all | grep compress
\nnas compressratio 1.00x -
\nnas compression lz4 local
\nnas refcompressratio 1.00x -
\nnas\/storage compressratio 1.00x -
\nnas\/storage compression lz4 local
\nnas\/storage refcompressratio 1.00x -
\nnas\/test compressratio 1.00x -
\nnas\/test compression lz4 inherited from nas
\nnas\/test refcompressratio 1.00x -<\/code><\/p>\nnas\/test compressratio 1.71x -
\nnas\/test compression lz4 inherited from nas
\nnas\/test refcompressratio 1.71x -<\/code><\/p>\n
\nThat though, is a topic for another day.<\/p>\n