ReWinD Blog

Last week, I replaced a drive in my RAIDZ that was starting to report errors. To be clear, I was actually getting errors on 2 or 3 drives, but most of them were on c7d0. When I was trying to see if I could hotswap out c7d0, I unplugged c8d0 and almost crapped my pants. When I realized how loose the SATA cables were in getting c8d0 back up and running in the array, I decided I better do this with the power off.

Here are the steps I took to make a backup and then rebuild my array. I decided to build the arrays a little differently, so these steps aren’t needed if you are just replacing a drive. “zpool replace storage c7d0″ with a new drive connected to controller 7 would have sufficed in my case.

To make a backup, I plugged in a 1TB USB drive and used fdisk to clear it into one big partition. I used the device driver manager in the Gnome GUI to tell me that it was sitting on c9t0d0. I created a ZFS filesystem named “backup” on it and then used “zpool send storage | zfs receive -f backup” to copy the filesystem to the other drive. This worked beautifully and only 12 hours or so later, I had a complete copy of all 900GB of data.

This points out the futility of backup for these large arrays. Even if you do have the drives, tape, or whatever, it’s hard to get more than 1 or 1.5GB per minute. That’s only 60GB to 90GB per hour. It takes a long time to backup a TB of data.

Once the backup was done, I used “zpool destroy -f storage” to break the array. I did this because I wanted to rebuild it as one 4 drive array with SATA drives and a single drive array with the other IDE drive in it. This way when I eventually upgrade to 1.5TB drives, I can just use four instead of the five drives that are in there now. With the drives ready to go, I created the new array and filesystem with the command “zpool create raidz storage c5d0 c6d0 c7d0 c8d0″. I also created a “movies” zpool for DVD rips that didn’t have to be redundant with the command “zpool create movies c4d0″.

Now that I had the two new arrays ready, I just used “cp” to copy the data back to the new arrays and put the folders in the right places (rips to /movies and everything important to /storage- the raidz array).

Project, Technology raidz, solaris, zfs

And no, this isn’t a comment on my server’s self-esteem. My RAIDZ pool is reporting that it’s got problems. It figures it happens on Friday the 13th. Weird, huh?

Actually I first saw the warnings a couple weeks ago when a scrub returned several errors in the zpool. The data was still good though because of checksumming in ZFS. I was getting a little anxious because I keep my iTunes library on the server along with raw footage from home movies. Important stuff to our family that I wouldn’t want to tell my wife that I had lost.

Most of the errors were on C7D0 (one of the four SATA drives in my setup) so I spent a week or two debating about whether I wanted to just upgrade the four 250GB drives to 1TB or 1.5TB drives. In the end, I decided on hoarding cash and I ordered a 250GB drive for $40. I’d still like to upgrade the array to 1.5TB drives when they hit $99 each, but I can wait to see if this new stimulus package will include free drives for homeowners along with the kickbacks to Wall St executives and help for stupid people who bought homes they can’t afford. I can only dream, right?

When the new drive arrived, I thought I would see if I could hot swap it. Smart, right? In hindsight it makes perfect sense that I would try testing out the resilency of ZFS and RAIDZ on a live array holding precious memories of my children’s births and early years when it was already starting to fail. Living life on the edge is what I like to do. Of course, when you try this on your own home file server, you might want to be a little more careful about tracing the SATA cables to make sure you actually pull the right one.

Panic. This was going to be a Friday the 13th for the ages.

There were a couple minutes there when I was literally sweating bullets. I was lucky though that the array wasn’t being used and so no data was lost. I brought c8d0 back up and cleared the zpool and all looked good.

Once I get everything up and running, I’ll write up the steps I used to make a backup, recreate the arrays, and copy my data back.

Project, Technology raidz, solaris, zfs

That was pretty easy. Here’s what I did…

• downloaded OpenSolaris 2008.11 Live CD
• burned ISO file to a CD
• booted fileserver from Live CD
• double-clicked “Install OpenSolaris” on desktop of Live CD
• answered the install questions (time zone, username, etc.)
• ran “Add More Software” which launches the package manager
• installed the “nas-server” collection
• rebooted (apparently some of the nas-server config files don’t load properly until a reboot)

Then from the command line, I did some magic to get my existing 5-drive RAIDZ pool mounted on the new OS. My zpool has the unimaginative name of “storage” which you will see in the following commands.

• zpool import -f storage
• zpool upgrade storage
• zfs upgrade storage

Now that my zpool was mounted and ready for the cifs service, I had to enter a few more commands to get cifs running.

• svcadm enable -r smb/server
• smbadm join -w <workgroup name>
• zfs set sharesmb=on storage
• sharemgr show -vp (just to check)

And that was that. I just love how simple these commands are now. A lot simpler than SAMBA, but we’ll see if the new cifs service is as stable and fast.

I still had to do a little chown and chmod work to get permissions right, but then everything was perfect. The mapped network drive on my PC fired right up without any changes, and the saved “connect to server” favorite on my Mac worked immediately as well.

I have iTunes set up so that its “iTunes Music” folder is on the fileserver. So now all is well there and I can continue to manage my library from my Mac laptop, but store all the music files on the server. I feel a little better knowing that RAIDZ is protecting all those media files from bitrot, but I still need to get some backup in place for my 500+GB media folder.

Next project will be to see if I can get Firefly running to share out my movies folder to the AppleTV.

Project, Technology cifs, iTunes, opensolaris, samba, smb, solaris

We had a power outage last night and the interruption has created some errors on disk0. I tried to fsck, but there’s not much use unless I boot from a live CD and umount the filesystem first. I decided that maybe this was a good time to go to Solaris 10 10/08 anyways.

But then I thought, if I’m going to reinstall, why not try OpenSolaris? The new 2008.11 has some nifty features and all I really care about is getting my ZFS pool shared out via SMB/CIFS anyways. Hmmm, and doesn’t OpenSolaris have built-in CIFS support? That could be fun to try…

So 30 minutes later, I’ve got the OpenSolaris ISO downloaded and the CD has been burned. Booted into the CD, and I’m immediately greeted (well not, immediately - booting from CD is slow) by a desktop with Firefox, etc. all ready to go. This is nice.

There’s a “Install OpenSolaris” icon on the desktop. I click that and it recognizes my 6 drives. I wiped the boot drive and now it’s chugging away. It’s about 50% done after what seems like 15 minutes or so (I’m catching up on Twitter while I wait, so who knows how long it’s really been). Can’t wait to see it in action and get my zpool mounted and CIFS working.

Here’s what I’m really excited about… Apparently there are a few people that have Firefly Media Server running on OpenSolaris. This is going to be great!

Project, Technology opensolaris, solaris

Check this thing out…

Yep. It actually solves the Rubik’s Cube using the light sensor. Awesome.

Uncategorized

Macy’s Thanksgiving Day Parade RickRoll

This is awesome. Thanks, Wil.

Uncategorized

Uncategorized

TheAppleBlog

Uncategorized

Uncategorized

Google seems to be backing down from their previous “index everything” stance taken with their book project and is spending more time partnering with copyright holders with sponsored channels on YouTube and now a joint-effort with LIFE to catalog all their images. They have about 20% of the collection scanned and should finish the rest over the next year.

Frankly, having access to this amazing collection of images is a perfect example of why I still like the Internet.

You can read the Google Blog post that explains the details here…

LIFE Photo Archive Available on Google Image Search

Technology google, LIFE, photography