zfs replace drive in raid10 (or other mirrored array) without hot spare

zfs replace drive in raid10 (or other mirrored array) without hot spare

First let’s see the failure:

zpool status -v

pool: pool0
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use ‘zpool clear’ to mark the device
repaired.
scan:
config:

NAME                        STATE     READ WRITE CKSUM
pool0                     DEGRADED     0     0     0
  mirror-0                  ONLINE       0     0     0
    wwn-0x5000c500645a168e  ONLINE       0     0     0
    wwn-0x5000c500648e648a  ONLINE       0     0     0
  mirror-1                  DEGRADED     0     0     0
    wwn-0x5000c5006459ebe2  ONLINE       0     0     0
    wwn-0x5000c500644ab516  FAULTED      2     0     0  too many errors
logs
  mirror-2                  ONLINE       0     0     0
    wwn-0x5001b448b5504c18  ONLINE       0     0     0
    wwn-0x5001b448b55067f3  ONLINE       0     0     0
cache
  wwn-0x5001b448b5504092    ONLINE       0     0     0
  wwn-0x5001b448b5102340    ONLINE       0     0     0

Note that I use the drives as they are listed in /dev/disk/by-id/. You can still use /dev/sd[letter] here instead, but if you are doing this, know that you should be using drive id’s instead.

Now let’s set the faulted drive offline and remove it(detach is the proper name for this):

zpool offline pool0 wwn-0x5000c500644ab516

zpool detach pool0 wwn-0x5000c500644ab516

zpool status -v

pool: pool0
state: ONLINE
scan:
config:

NAME                        STATE     READ WRITE CKSUM
pool0                       ONLINE       0     0     0
  wwn-0x5000c5006459ebe2    ONLINE       0     0     0
  mirror-1                  ONLINE       0     0     0
    wwn-0x5000c500645a168e  ONLINE       0     0     0
    wwn-0x5000c500648e648a  ONLINE       0     0     0
logs
  mirror-2                  ONLINE       0     0     0
    wwn-0x5001b448b5504c18  ONLINE       0     0     0
    wwn-0x5001b448b55067f3  ONLINE       0     0     0
cache
  wwn-0x5001b448b5504092    ONLINE       0     0     0
  wwn-0x5001b448b5102340    ONLINE       0     0     0

OK, at this point the array is still working and we are good.

zpool attach pool0 wwn-0x5000c5006459ebe2 wwn-0x5000c5004fc868cd

zpool status -v

pool: pool0
state: ONLINE
scan:
config:

NAME                        STATE     READ WRITE CKSUM
hybrid0                     ONLINE       0     0     0
  mirror-0                  ONLINE       0     0     0
    wwn-0x5000c500645a168e  ONLINE       0     0     0
    wwn-0x5000c500648e648a  ONLINE       0     0     0
  mirror-1                  ONLINE       0     0     0
    wwn-0x5000c5006459ebe2  ONLINE       0     0     0
    wwn-0x5000c5004fc868cd  ONLINE       0     0     0
logs
  mirror-2                  ONLINE       0     0     0
    wwn-0x5001b448b5504c18  ONLINE       0     0     0
    wwn-0x5001b448b55067f3  ONLINE       0     0     0
cache
  wwn-0x5001b448b5504092    ONLINE       0     0     0
  wwn-0x5001b448b5102340    ONLINE       0     0     0

So after any resilvering, you can see that the new drive, wwn-0x5000c5004fc868cd, replaced wwn-0x5000c500644ab516 in the mirror and the raid10 array is still functional.

Before doing this, you SHOULD ALWAYS HAVE A COMPLETE BACKUP!