View source for Replacing a failed disk in a ZFS/ZPOOL raid array

This has been used on a SPARC S7-2, but should be relevant to most modern sparc based servers:

1. Identify the failed disk in the zpool array:

  # '''zpool status'''
    pool: rpool
   state: DEGRADED
   
  config:
  
          NAME                       STATE      READ WRITE CKSUM
          rpool                      DEGRADED      0     0     0
            mirror-0                 DEGRADED      0     0     0
              c0t5000CCA02F613ACCd0  ONLINE        0     0     0
              '''c0t5000CCA02D0F6C44d0'''  UNAVAIL       0     0     0
 
IF the drive is not fully faulted/"UNAVAIL", like this:

  # zpool  status
    pool: rpool
   state: DEGRADED
  
  config:
  
          NAME                       STATE      READ WRITE CKSUM
          rpool                      DEGRADED      0     0     0
            mirror-0                 DEGRADED      0     0     0
              c0t5000CCA02F613ACCd0  ONLINE        0     0     0
              '''c0t5000CCA02D0F6C44d0'''  DEGRADED      0     0     0

THEN you will need to OFFLINE the disk first, like so:

  '''zpool offline rpool''' c0t5000CCA02D0F6C44d0

2. Find the device path for the drive you want to remove:

  # '''diskinfo D:devchassis-path'''
  D:devchassis-path                   c:occupant-compdev
  ----------------------------------  ---------------------
  /dev/chassis/SYS/HDD0/disk          c0t5000CCA02F613ACCd0
  /dev/chassis/SYS/HDD1/disk          '''c0t5000CCA02D0F6C44d0'''
  /dev/chassis/SYS/HDD2               -
  /dev/chassis/SYS/HDD3               -
  /dev/chassis/SYS/HDD4               -
  /dev/chassis/SYS/HDD5               -
  /dev/chassis/SYS/HDD6               -
  /dev/chassis/SYS/HDD7               -
  /dev/chassis/SYS/MB/EUSB_DISK/disk  c1t0d0

3. Check the drive's status:

  # '''cfgadm -al'''
  Ap_Id                          Type         Receptacle   Occupant     Condition
  /SYS/DBP/NVME0                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME1                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME2                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME3                 unknown      empty        unconfigured unknown
  c3                             scsi-sas     connected    configured   unknown
  c3::w5000cca02f613acd,0        disk-path    connected    configured   unknown
  c4                             scsi-sas     connected    configured   unknown
  '''c4::w5000CCA02D0F6C45,0        disk-path    connected    configured   unknown'''
  usb0/1                         usb-storage  connected    configured   ok
  usb0/2                         usb-hub      connected    configured   ok

'''NOTICE''' the slight mismatch between the output of the last two commands.

4. Unconfigure the drive:

  # '''cfgadm -c unconfigure c4::w5000CCA02D0F6C45,0''' 

and check that it worked:

  # '''cfgadm -al'''
  Ap_Id                          Type         Receptacle   Occupant     Condition
  /SYS/DBP/NVME0                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME1                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME2                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME3                 unknown      empty        unconfigured unknown
  c3                             scsi-sas     connected    configured   unknown
  c3::w5000cca02f613acd,0        disk-path    connected    configured   unknown
  c4                             scsi-sas     connected    configured   unknown
  c4::w5000CCA02D0F6C45,0        disk-path    connected    '''unconfigured'''   unknown

IF, when you try to unconfigure, you get this error:

  # cfgadm -c unconfigure c4::w5000CCA02D0F6C45,0
  '''cfgadm: Hardware specific failure: failed to unconfigure SCSI device: I/O error'''

Go back to step one and make the disk '''offline''' in the ZFS pool

5. Turn on the Ok to Remove indicator for that drive:

  # '''fmadm set-indicator /dev/chassis/SYS/HDD1/disk ok2rm on'''

and check that it worked:

  # '''fmadm get-indicator /dev/chassis/SYS/HDD1/disk ok2rm'''
  The indicator (ok2rm) is set to on.

6. Remove the failed drive and replace with the new one.

7. The new drive should be configured automatically, but check it anyway:

  # '''cfgadm -al'''
  Ap_Id                          Type         Receptacle   Occupant     Condition
  /SYS/DBP/NVME0                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME1                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME2                 unknown      empty        unconfigured unknown
  /SYS/DBP/NVME3                 unknown      empty        unconfigured unknown
  c3                             scsi-sas     connected    configured   unknown
  c3::w5000cca02f613acd,0        disk-path    connected    configured   unknown
  c4                             scsi-sas     connected    configured   unknown
  c4::w5000cca07d293ca1,0        disk-path    connected    '''configured'''   unknown
  usb0/1                         usb-storage  connected    configured   ok
  usb0/2                         usb-hub      connected    configured   ok

If not, configure with:

  # '''cfgadm -c unconfigure c4::w5000cca07d293ca1,0''' 

8. Get the name od the new drive from the format command:

  # '''format'''
  Searching for disks...done
  
  
  AVAILABLE DISK SELECTIONS:
         0. c0t5000CCA02F613ACCd0 <HGST-H101860SFSUN600G-A990-558.91GB>
            /scsi_vhci/disk@g5000cca02f613acc
            /dev/chassis/SYS/HDD0/disk
         1. '''c0t5000CCA07D293CA0d0''' <HGST-H101860SFSUN600G-A990-558.91GB>
            /scsi_vhci/disk@g5000cca07d293ca0
            /dev/chassis/SYS/HDD1/disk
         2. c1t0d0 <VT-eUSB-7722-1.91GB>
            /pci@300/pci@1/pci@0/pci@2/usb@0/storage@1/disk@0,0
            /dev/chassis/SYS/MB/EUSB_DISK/disk
  Specify disk (enter its number):

9. Replace the failed disk with the new disk in the zpool:

  # '''zpool replace rpool c0t5000CCA02D0F6C44d0 c0t5000CCA07D293CA0d0'''

10. Check the progress with:

  # '''zpool status'''
    pool: rpool
   state: ONLINE
    scan: resilvered 75.8G in 8m24s with 0 errors on Wed Feb 27 11:39:38 2019
  
  config:
  
          NAME                       STATE      READ WRITE CKSUM
          rpool                      ONLINE        0     0     0
            mirror-0                 ONLINE        0     0     0
              c0t5000CCA02F613ACCd0  ONLINE        0     0     0
              c0t5000CCA07D293CA0d0  ONLINE        0     0     0
  
  errors: No known data errors

11. Once the zpool is happy again, install the boot block on the new disk:

  # '''installboot -f -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t5000CCA07D293CA0d0s0'''