Friday, March 11, 2022

TrueNAS Scale and ZFS Write Throttling

By default on OpenZFS/Linux the max amount of dirty data is capped at 4GB. This means that if your nas server is connected over network with faster throughput than your pool can sustain (writes), you will quickly end up being throttled when writing lots of data (>>4GB). This can easily happen if your home nas server is connected over 10GbE and its internal pool configuarion can sustain much less (e.g. 2 mirrored disks will sustain less than 200MB/s of writes) and you are writing from time to time large amounts of data.

In such a case you might potentially improve performance significantly by tuning ZFS write throttling.

Example workflow is - you are editing large'ish video files on your workstation (also connected to 10GbE) on your loval NVMEs drives. Once you are done you want to make a copy to your nas server - the files lets say are 50GB in total. You will not be able to write at full network speed for the whole transfer due to write throttling.

If your nas server has plenty of RAM, you could increase the dirty data threshold - if you can increase it to a larger value then the max amount of data you will be writing in one session, then the end result should be that you will sustain a very high throughput over network and from your perspective the whole transfer will be significantly quicker while the nas server will be destaging the data to disks in background.

Let's say you want to increase it to 100GB, to do so:
    # echo 107374182400 >/sys/module/zfs/parameters/zfs_dirty_data_max
There are pros and cons obviously, depending on specifc situation. For example if you have multiple clients writing to your NAS server then increasing the threshold might not be such a good idea. Generally, if you only have one client doing any heave writes from time to time you should benefit from this tuning.

There is an excellent blog on tuning zfs write throttle if you want to understand it in more detail.

How do you make the setting persitent in TrueNAS Scale in a supported way?
Go to System Settings -> Advanced -> Init/Shutdown Scripts -> Add
and populate it with:
  Type: Command
  Command: echo 107374182400 >/sys/module/zfs/parameters/zfs_dirty_data_max
  When: Post Init
After reboot it will pick up the new value.
The advantage of setting it up this way is that it will be included in exported TrueNAS config file.

Wednesday, February 09, 2022

TrueNAS Scale & ZFS Wrapping Key

 I've been playing with TrueNAS scale recently and while the BUI allows you to export/download ZFS wrapping key I wanted to know how to get the key manually. After a quick look at the code I found that the key is stored in sqlite db kept on root file system.

root@truenas[~]# sqlite3 /data/freenas-v1.db
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select * from storage_encrypteddataset;
1|backup|ioI/B72PEllUJjumWpWHkdhDDCd2l2eopFEJgWYIpcAcTT1v0NyYicjzKiHfuoncL2Mklfa45pUJIyxzGFGobr17b1HtprjSth/X9yyfsnROCK/xQL+SVmO/5fT/KabfSSiz8+IfDH8=|
The key itself is encrypted so you need to decrypt it first before it can be used with ZFS. A simple python script to do it attached below.
root@truenas[~]# ./decode_key.py
dataset: backup
  key: 16f7677b514ef39bc162312274c76da24221ecc5a2f01e6ba0bhfeec054d9162
(both the encrypted and decrypted keys above have been modified for this blog entry)
root@truenas[~]# cat decode_key.py
#!/usr/bin/python3

# based on /usr/lib/migrate113/freenasUI/system/migrations/0022_cloud_sync.py

import sys
import base64
from Cryptodome.Cipher import AES
import sqlite3


PWENC_BLOCK_SIZE = 32
PWENC_FILE_SECRET = '/data/pwenc_secret'
PWENC_PADDING = b'{'


def pwenc_get_secret():
    with open(PWENC_FILE_SECRET, 'rb') as f:
        secret = f.read()
    return secret


def pwenc_decrypt(encrypted=None):
    if not encrypted:
        return ""
    from Cryptodome.Util import Counter
    encrypted = base64.b64decode(encrypted)
    nonce = encrypted[:8]
    encrypted = encrypted[8:]
    cipher = AES.new(
        pwenc_get_secret(),
        AES.MODE_CTR,
        counter=Counter.new(64, prefix=nonce),
    )
    return cipher.decrypt(encrypted).rstrip(PWENC_PADDING).decode('utf8')


if len(sys.argv) == 2:
    print(pwenc_decrypt(sys.argv[1]))
    exit(0)

dbcon = sqlite3.connect('/data/freenas-v1.db')
dbcur = dbcon.cursor()
for row in dbcur.execute('select * from storage_encrypteddataset'):
    ds_id, ds_name, ds_enc_key, kmip_enc_key = row
    #print(ds_id, ds_name, ds_enc_key, pwenc_decrypt(ds_enc_key))
    print(f'dataset: {ds_name}\n  key: {pwenc_decrypt(ds_enc_key)}\n')