ZFS: Device Removal

As finally publicly presented at Solaris Tech Day in Vienna couple of weeks ago, ZFS in Solaris 11.4 will have the long awaited on-line device removal feature. This is top-level vdev removal only, but still very useful in some scenarios.

Here is an example on how it works.

First, let's create a test pool whish is a mirror of two disks:

root@solaris:~# zpool create test mirror c1t1d0 c1t3d0
root@solaris:~# zpool status test
  pool: test
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        test        ONLINE        0     0     0
          mirror-0  ONLINE        0     0     0
            c1t1d0  ONLINE        0     0     0
            c1t3d0  ONLINE        0     0     0

errors: No known data errors

Now, let's "accidently" add a single disk to stripe with the mirror and copy some data into the pool:

root@solaris:~# zpool add -f test c1t4d0
root@solaris:~# zpool status test
  pool: test
 state: ONLINE
  scan: none requested
config:

        NAME        STATE      READ WRITE CKSUM
        test        ONLINE        0     0     0
          mirror-0  ONLINE        0     0     0
            c1t1d0  ONLINE        0     0     0
            c1t3d0  ONLINE        0     0     0
          c1t4d0    ONLINE        0     0     0

errors: No known data errors

root@solaris:~# cp -rp /usr/share/doc /test/
^C

root@solaris:~# gdf -h /test
Filesystem      Size  Used Avail Use% Mounted on
test            2.0G  375M  1.6G  19% /test

root@solaris:~# zpool iostat -v test
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
test         375M  1.60G      0    181  4.12K  5.04M
  mirror-0   242M   766M      0    173    203  3.23M
    c1t1d0      -      -      0     16  7.73K  3.28M
    c1t3d0      -      -      0     16  7.53K  3.28M
  c1t4d0     132M   876M      0      9  4.91K  2.26M
----------  -----  -----  -----  -----  -----  -----

Now, if we want to remove the accidently added disk drive it is trivial to do so:

root@solaris:~# zpool remove test c1t4d0

And let's check pool status after the device was removed:

root@solaris:~# zpool status test
  pool: test
 state: ONLINE
  scan: resilvered 132M in 1s with 0 errors on Fri Mar 30 01:53:17 2018

config:

        NAME                      STATE      READ WRITE CKSUM
        test                      ONLINE        0     0     0
          mirror-0                ONLINE        0     0     0
            c1t1d0                ONLINE        0     0     0
            c1t3d0                ONLINE        0     0     0

errors: No known data errors

root@solaris:~# zpool iostat -v test
                             capacity     operations    bandwidth
pool                      alloc   free   read  write   read  write
------------------------  -----  -----  -----  -----  -----  -----
test                       378M   630M      2    105  8.66K  1.63M
  mirror-0                 378M   630M      2     80  5.14K  1.21M
    c1t1d0                    -      -      0      8  5.09K  1.22M
    c1t3d0                    -      -      1      7  5.58K  1.22M
------------------------  -----  -----  -----  -----  -----  -----