I have a ZFS pool containing 8 disks (4×2TiB + 2×4TiB + 2×4TiB) on an 8 bay server:

 state: DEGRADED
status: ....
config:

        NAME           STATE     READ WRITE CKSUM
        ${SRC_POOL}    DEGRADED     0     0     0
          raidz2-0     DEGRADED     0     0     0
            da0p2      ONLINE       0     0     0
            da1p2      DEGRADED     0     0     0
            da2p2      ONLINE       0     0     0
            da3p2      ONLINE       0     0     0
          mirror-1     ONLINE       0     0     0
            da4p2      ONLINE       0     0     0
            da5p2      ONLINE       0     0     0
          mirror-2     ONLINE       0     0     0
            da6p2      ONLINE       0     0     0
            da7p2      ONLINE       0     0     0

errors: No known data errors

When it comes to the time to replace the warning drive and expand the storage pool, I want to give up the current striped pool and create a new pool containing a single RAID-Z2 vdev (4×8TiB). Now here is the problem:

How to migrate create a new pool with all 8 bays are occupied?

The traditional way is to backup the content to external storage and restore. But here is a possible solution without any external storage: creating a degraded pool with minimum devices.

Tricky Drive Replacement

At least 2 online devices are required to keep a RAID-Z2 vdev working. So the trick is to pull out 2 drives from the original pool and insert 2 new blank drives.

+------+------+------+------+     +------+------+------+------+
|        RAID-Z2 4×2T       |     | NEW1 | NEW2 |RAID-Z2 2×2T |
|------+------+------+------| ==> |------+------+------+------|
| MIRROR 2×4T | MIRROR 2×4T |     | MIRROR 2×4T | MIRROR 2×4T |
+------+------+------+------+     +------+------+------+------+

Now the original pool enters DEGRADED status (it has been in degraded status anyway).

Create a Degraded Pool

Create some sparse files as "fake devices". The size of the sparse files should match the size of the hard drives (for any convenience).

SPARSE_SIZE=8T
SPARSE_FP_1=/root/sparsefile1
SPARSE_FP_2=/root/sparsefile2
truncate -s ${SPARSE_SIZE} ${SPARSE_FP_1}
truncate -s ${SPARSE_SIZE} ${SPARSE_FP_2}
# ls -l /root/sparsefile*
-rw-r--r--  1 root  wheel   8.0T Oct 26 04:15 sparsefile1
-rw-r--r--  1 root  wheel   8.0T Oct 26 04:15 sparsefile2

Create a ZFS Storage Pool at RAID Z2 with 2 hard drives and 2 sparse files:

zpool create ${POOL_NAME} raidz2 \
	da0p2 da1p2 \
    ${SPARSE_FP_1} \
    ${SPARSE_FP_2}
zfs set aclmode=passthrough ${POOL_NAME}
zfs set compression=lz4 ${POOL_NAME}

Set the sparse files to offline so that no actual data will be written:

zpool offline ${POOL_NAME} ${SPARSE_FP_1}
zpool offline ${POOL_NAME} ${SPARSE_FP_2}
root@storage-01:~ # zpool status SKG_STORAGE_1
  pool: ${POOL_NAME}
 state: DEGRADED
status: ....
config:

        NAME                                            STATE     READ WRITE CKSUM
        ${POOL_NAME}           DEGRADED     0     0     0
          raidz2-0             DEGRADED     0     0     0
            da0p2              ONLINE       0     0     0
            da1p2              ONLINE       0     0     0
            /root/sparsefile1  OFFLINE      0     0     0
            /root/sparsefile2  OFFLINE      0     0     0

errors: No known data errors

Now we have a degraded pool with 2 actual drives and 2 fake drives. Finally adjust export the pool so that it could be imported in FreeNAS Web UI:

zpool export ${POOL_NAME}

Take and Send Snapshots

Create a snapshot:

zfs snapshot -r ${SRC_POOL}@migration-base

Dry-run before sending the actual data and proceed if everything is checked:

zfs send -Rvn -i ${SRC_POOL}@migration_base
zfs send -Rv -i snapshot ${SRC_POOL}@migration-base | pv | zfs receive -Fsvd ${POOL_NAME}@migration-base

Replace Fake Devices with Real Ones

zpool replace ${POOL_NAME} ${SPARSE_FP_1} da3p2
zpool replace ${POOL_NAME} ${SPARSE_FP_2} da4p2

References