TrueNAS Scale and ZFS Write Throttling

By default on OpenZFS/Linux the max amount of dirty data is capped at 4GB. This means that if your nas server is connected over network with faster throughput than your pool can sustain (writes), you will quickly end up being throttled when writing lots of data (>>4GB). This can easily happen if your home nas server is connected over 10GbE and its internal pool configuarion can sustain much less (e.g. 2 mirrored disks will sustain less than 200MB/s of writes) and you are writing from time to time large amounts of data.

In such a case you might potentially improve performance significantly by tuning ZFS write throttling.

Example workflow is - you are editing large'ish video files on your workstation (also connected to 10GbE) on your loval NVMEs drives. Once you are done you want to make a copy to your nas server - the files lets say are 50GB in total. You will not be able to write at full network speed for the whole transfer due to write throttling.

If your nas server has plenty of RAM, you could increase the dirty data threshold - if you can increase it to a larger value then the max amount of data you will be writing in one session, then the end result should be that you will sustain a very high throughput over network and from your perspective the whole transfer will be significantly quicker while the nas server will be destaging the data to disks in background.

Let's say you want to increase it to 100GB, to do so:

    # echo 107374182400 >/sys/module/zfs/parameters/zfs_dirty_data_max

There are pros and cons obviously, depending on specifc situation. For example if you have multiple clients writing to your NAS server then increasing the threshold might not be such a good idea. Generally, if you only have one client doing any heave writes from time to time you should benefit from this tuning.

There is an excellent blog on tuning zfs write throttle if you want to understand it in more detail.

How do you make the setting persitent in TrueNAS Scale in a supported way?
Go to System Settings -> Advanced -> Init/Shutdown Scripts -> Add

and populate it with:

  Type: Command
  Command: echo 107374182400 >/sys/module/zfs/parameters/zfs_dirty_data_max
  When: Post Init

After reboot it will pick up the new value.
The advantage of setting it up this way is that it will be included in exported TrueNAS config file.