Testing ZFS/L2ARC

Solaris 11.4 - setting zfs_arc_collect_check=0 via mdb (takes immediate effect) or via /etc/system makes ZFS to start feeding l2arc immediately. Notice that this can negatively impact ARC performance so use it with care. This is useful for testing if you want to push some data to into L2ARC quicker/sooner, especially on large memory systems. The variable is checked by arc_can_collect() function (if it returns 1 then l2arc cab be fed, if zero it can't).

DTrace stop() action

The stop() action in DTrace stops an entire process... well, actually it doesn't. It stops a single thread in a multi-threaded process, which got me surprised as I always thought it did stop an entire process. Now, this is actually very useful, though a stopall() action which would stop all threads could now be useful as well :)

Update: this is getting more complicated now, the way stop() action behaves depends on probe type it is called from. For example, if called from a probe from syscall provider it will just stop a thread which called the syscall, but if called from a probe from PID provider it will stop entire process with all its threads. This is getting confusing...

btw: pstop PID stops entire process while pstop PID/LWPID stops a single thread

DTrace %Y print format with nanoseconds

Small but useful extension to DTrace is now available in Solaris 11.4.SRU6. It allows to easily print current date with an optional nanosecond resolution. It is disabled by default for backward compatibility.

To enable it you need to add timedecimals option to dtrace:


# dtrace -q -x timedecimals -n syscall::open*:entry \
    '{printf("%Y %s called %s()\n", walltimestamp, execname, probefunc);}'
2019 Mar  1 11:50:48.774114445 firefox called openat64()
2019 Mar  1 11:50:49.149290513 dtrace called openat()
2019 Mar  1 11:50:49.149283375 dtrace called openat()
2019 Mar  1 11:50:50.030217373 firefox called openat64()
2019 Mar  1 11:50:49.974253263 firefox called openat64()
2019 Mar  1 11:50:50.114684381 VBoxService called openat()
^C

You can also specify number of decimal places to be printed, fox example:

# dtrace -q -x timedecimals=2 -n syscall::open*:entry \
    '{printf("%Y %s called %s()\n", walltimestamp, execname, probefunc);}'
2019 Mar  1 11:56:51.09 VBoxService called openat()
2019 Mar  1 11:56:51.09 VBoxService called openat()
2019 Mar  1 11:56:51.45 dtrace called openat()
^C

RAID-Z improvements and cloud device support

Solaris 11.4 introduced few new ZFS pool versions with interesting new features or enhancements:

# zpool upgrade -v
...
 38  Xcopy with encryption
 39  Resilver restart enhancements
 40  New deduplication support
 41  Asynchronous dataset destroy
 42  Reguid: ability to change the pool guid
 43  RAID-Z improvements and cloud device support
 44  Device removal
...

The RAID-Z improvements mean that data is written more efficiently - in some cases it can now store more data in a pool than before. But even more importantly the performance (both throughput and IOPS) of RAIDZ is now close to RAID10!

Spectre and Kernel Modules

On Linux one needs to recompile kernel modules to get protection, while on Solaris this is not necessary. Once you are on Solaris 11.4 with Spectre fixes enabled, all kernel modules, even compiled on older Solaris releases, just work and are protected. Nothing special to do there.

bpftrace

Right, finally Linux is getting something similar and useful to DTrace, see bpftrace. However for it to be useful in enterprise it has to be included in RedHat - I wonder how long it will take though... but maybe around 2020 this will finally happen and then  Linux will truly have an equivalent of DTrace, even if 15 years later.

Solaris: Spectre v2 & Meltdown fixes

Solaris 11.4 includes fixes for Meltdown and Spectre v2 (fixes for v1 were delievered few months ago for 11.3 via SRU and are also included in 111.4). What I really like about them is that you can turn them on or off via sxadm. The sxadm command will also report if your HW requires the fixes and if they are enabled or not. Additionally there is an FMA alert generated if you HW should have fixes enabled but due to old microcode it can't be done - so this way you also get alerting. Very nice intergration indeed.

Example output with Solaris running in Virtual Box:

# sxadm status
EXTENSION           STATUS                        FLAGS
aslr                enabled (tagged-files)        u-c--
nxstack             enabled (all)                 u-c--
nxheap              enabled (tagged-files)        u-c--
kpti                enabled                       -kcr-
ibpb                not supported                 -----
ibrs                not supported                 -----
smap                not supported                 -----

The kpti is fix for Meltdow and it is active, while ibpb and ibrs are mitigations for Spectre v2 and are not enabled as these are not supported on this HW. Let's see how FMA is reporting an old version of microcode: 

# fmadm faulty
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Oct 02 14:19:24 383538f1-9268-4a07-9ff8-86be48c02e72  SUNOS-8000-LG  Major    

Problem Status            : open
Diag Engine               : software-diagnosis / 0.2
System
    Manufacturer          : unknown
    Name                  : unknown
    Part_Number           : unknown
    Serial_Number         : unknown

System Component
    Manufacturer          : innotek GmbH
    Name                  : VirtualBox
    Part_Number           : 
    Serial_Number         : 0
    Firmware_Manufacturer : innotek GmbH
    Firmware_Version      : (BIOS)VirtualBox
    Firmware_Release      : (BIOS)12.01.2006
    Host_ID               : 00482293
    Server_Name           : solaris

----------------------------------------
Suspect 1 of 1 :
   Problem class : alert.oracle.solaris.cpu.firmware.security
   Certainty   : 100%

   FRU
     Status           : Active
     Location         : "/SYS/MB"
     Manufacturer     : unknown
     Name             : unknown
     Part_Number      : unknown
     Revision         : unknown
     Serial_Number    : unknown
     Chassis
        Manufacturer  : Oracle Corporation
        Name          : VirtualBox
        Part_Number   : 
        Serial_Number : 0
   Resource
     Status           : Active

Response    : No automated response available

Impact      : Oracle Solaris is not running with Spectre Vulnerability
              Mitigation Enabled

Action      : Update the CPU with Spectre capable microcode. Please refer to
              the associated reference document at
              http://support.oracle.com/msg/SUNOS-8000-LG for the latest
              service procedures and policies regarding this diagnosis.

Solaris 11.4 GA

Solaris 11.4 GA is finally out and available for download.
See What’s New

dumpadm -d none

Solaris 11.3 still doesn't support dumpadm -d none. This is useful in some scenarios, for example when troubleshooting failed AI installations when you try to restart it withouth rebooting. This will generally fail as it won't be able to destroy rpool as there is a dump device already configured there. 

There is a workaround though.

Edit /etc/dumpadm.conf file and comment out line containing DUMPADM_DEVICE, then run dumpadm -u. This will unconfigure dump device entirely. Then just run zpool destroy rpool and now you can svcadm clear auto-installer.

ZFS Raw Send

This got finally integrated into 11.3 SRU 11.3.33.5.0 zfs send compressed data (Bug 15387669)

Extracting ZFS Wrapping Key

Thanks to Victor Latushkin, below is a procedure on how to extract ZFS wrapping key for a given dataset. Notice that key length is encoded in number of bits so it needs to be converted.

# cat /tmp/p
aaaaaaaaaabbbbbbbbbbccccccccccz

# zfs create -o encryption=aes-256-ccm -o keysource="raw,file:///tmp/p" test-0/testp
# zfs get objsetid test-0/testp
NAME                              PROPERTY  VALUE  SOURCE
test-0/testp  objsetid  192    -
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix zvpsm scsi_vhci iommu zfs lsc sd ip hook neti arp usba kssl stmf stmf_sbd sockfs lofs random idm nvme sata ufs cpc crypto fcip fctl fcp smbsrv nfs zvmm logindmux
nsmb ptm sppp ipc ]
> ::spa
ADDR                 STATE NAME                   REALNAME
ffffa1c13a33b000    ACTIVE test-0 -
ffffa1c13a38b000    ACTIVE test-1 -
ffffa1c13a097000    ACTIVE rpool                  -
> ffffa1c13a33b000::spa |::print spa spa_keystore->sk_dslkeys|::walk avl|::if zcrypt_keystore_node_t skn_os = 0t192|::print zcrypt_keystore_node_t skn_wrapkey->zk_key.cku_data.cku_key_value.cku_v_length
skn_wrapkey->zk_key.cku_data.cku_key_value.cku_v_length = 0x100
> 0x100 % 8 = X
                20
> ffffa1c13a33b000::spa |::print spa spa_keystore->sk_dslkeys|::walk avl|::if zcrypt_keystore_node_t skn_os = 0t192|::print zcrypt_keystore_node_t skn_wrapkey->zk_key.cku_data.cku_key_value.cku_v_data|::dump -un 0x20
                    0 1 2 3  4 5 6 7  8 9 a b  c d e f  0123456789abcdef
ffffa1c149dcdd20:  61616161 61616161 61616262 62626262  aaaaaaaaaabbbbbb
ffffa1c149dcdd30:  62626262 63636363 63636363 63637a0a  bbbbccccccccccz.

Utilizing CTF in MDB from a binary

Having CTF compiled into binaries is very useful for debugging. Recently I wanted to print some structures by utilizing mdb and ctf from a binary which is not running and I didn't have a core file either. Helpful folks from Oracle suggested the below nice workaround:

# mdb /path/to/bin
> main:b
> ::run
> ::print -at rx_call
...

It sets a breakpoint at main(), then executes it but it stops when calling main(), at this stage CTF data is already loaded by linker.

Support for CTF in Userland in DTrace

I wish Solaris 11 supported CTF in DTrace in user-land in a similar manner as Illumos does.

11.3.SRU31 - updated pam_list

The just released Solaris 11.3 SRU31 has an updated pam_list module which adds support for '*' and comments. The '*' wildcard is really useful, as it allows common PAM configuration where access to a server can be managed only by an allow file. For example, in /etc/pam.d/XXX you can now have:

account sufficient pam_list.so.1 allow=/etc/security/access.conf

If the access.conf file has only '*' which means all users have access, or you can just list users, netgroups or unixgroups.

To achieve the '*' before one had to modify the PAM configuration or use a different module (for example compile pam_access from Linux).

This is a good example of one of the small but very useful changes.

GCC 7 on Solaris 11.4

How to get gcc-7 on Solaris 11.4?

root@solaris:~# pkg install gcc-7
           Packages to install: 14
           Mediators to change:  1
            Services to change:  1
       Create boot environment: No
Create backup boot environment: No

DOWNLOAD                      PKGS         FILES    XFER (MB)   SPEED
Completed                    14/14     1822/1822  332.0/332.0  423k/s

PHASE                                          ITEMS
Installing new actions                     2253/2253
Updating package state database                 Done 
Updating package cache                           0/0 
Updating image state                            Done 
Creating fast lookup database                   Done 
Updating package cache                           1/1 
root@solaris:~# 

root@solaris:~# gcc --version
gcc (GCC) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@solaris:~# 

Older versions are available as well.