Last week, I replaced a drive in my RAIDZ that was starting to report errors. To be clear, I was actually getting errors on 2 or 3 drives, but most of them were on c7d0. When I was trying to see if I could hotswap out c7d0, I unplugged c8d0 and almost crapped my pants. When I realized how loose the SATA cables were in getting c8d0 back up and running in the array, I decided I better do this with the power off.
Here are the steps I took to make a backup and then rebuild my array. I decided to build the arrays a little differently, so these steps aren’t needed if you are just replacing a drive. “zpool replace storage c7d0″ with a new drive connected to controller 7 would have sufficed in my case.
To make a backup, I plugged in a 1TB USB drive and used fdisk to clear it into one big partition. I used the device driver manager in the Gnome GUI to tell me that it was sitting on c9t0d0. I created a ZFS filesystem named “backup” on it and then used “zpool send storage | zfs receive -f backup” to copy the filesystem to the other drive. This worked beautifully and only 12 hours or so later, I had a complete copy of all 900GB of data.
This points out the futility of backup for these large arrays. Even if you do have the drives, tape, or whatever, it’s hard to get more than 1 or 1.5GB per minute. That’s only 60GB to 90GB per hour. It takes a long time to backup a TB of data.
Once the backup was done, I used “zpool destroy -f storage” to break the array. I did this because I wanted to rebuild it as one 4 drive array with SATA drives and a single drive array with the other IDE drive in it. This way when I eventually upgrade to 1.5TB drives, I can just use four instead of the five drives that are in there now. With the drives ready to go, I created the new array and filesystem with the command “zpool create raidz storage c5d0 c6d0 c7d0 c8d0″. I also created a “movies” zpool for DVD rips that didn’t have to be redundant with the command “zpool create movies c4d0″.
Now that I had the two new arrays ready, I just used “cp” to copy the data back to the new arrays and put the folders in the right places (rips to /movies and everything important to /storage- the raidz array).