Archive for the ‘Project’ Category

Backing up and Restoring a RAIDZ Array

February 20th, 2009 No comments

Last week, I replaced a drive in my RAIDZ that was starting to report errors. To be clear, I was actually getting errors on 2 or 3 drives, but most of them were on c7d0. When I was trying to see if I could hotswap out c7d0, I unplugged c8d0 and almost crapped my pants. When I realized how loose the SATA cables were in getting c8d0 back up and running in the array, I decided I better do this with the power off.

Here are the steps I took to make a backup and then rebuild my array. I decided to build the arrays a little differently, so these steps aren’t needed if you are just replacing a drive. “zpool replace storage c7d0″ with a new drive connected to controller 7 would have sufficed in my case.

To make a backup, I plugged in a 1TB USB drive and used fdisk to clear it into one big partition. I used the device driver manager in the Gnome GUI to tell me that it was sitting on c9t0d0. I created a ZFS filesystem named “backup” on it and then used “zpool send storage | zfs receive -f backup” to copy the filesystem to the other drive. This worked beautifully and only 12 hours or so later, I had a complete copy of all 900GB of data.

This points out the futility of backup for these large arrays. Even if you do have the drives, tape, or whatever, it’s hard to get more than 1 or 1.5GB per minute. That’s only 60GB to 90GB per hour. It takes a long time to backup a TB of data.

Once the backup was done, I used “zpool destroy -f storage” to break the array. I did this because I wanted to rebuild it as one 4 drive array with SATA drives and a single drive array with the other IDE drive in it. This way when I eventually upgrade to 1.5TB drives, I can just use four instead of the five drives that are in there now. With the drives ready to go, I created the new array and filesystem with the command “zpool create raidz storage c5d0 c6d0 c7d0 c8d0″. I also created a “movies” zpool for DVD rips that didn’t have to be redundant with the command “zpool create movies c4d0″.

Now that I had the two new arrays ready, I just used “cp” to copy the data back to the new arrays and put the folders in the right places (rips to /movies and everything important to /storage- the raidz array).

Categories: Project, Technology Tags: , ,

My RAIDZ Storage Pool is Degraded

February 13th, 2009 No comments

And no, this isn’t a comment on my server’s self-esteem. My RAIDZ pool is reporting that it’s got problems. It figures it happens on Friday the 13th. Weird, huh?

Actually I first saw the warnings a couple weeks ago when a scrub returned several errors in the zpool. The data was still good though because of checksumming in ZFS. I was getting a little anxious because I keep my iTunes library on the server along with raw footage from home movies. Important stuff to our family that I wouldn’t want to tell my wife that I had lost.

Most of the errors were on C7D0 (one of the four SATA drives in my setup) so I spent a week or two debating about whether I wanted to just upgrade the four 250GB drives to 1TB or 1.5TB drives. In the end, I decided on hoarding cash and I ordered a 250GB drive for $40. I’d still like to upgrade the array to 1.5TB drives when they hit $99 each, but I can wait to see if this new stimulus package will include free drives for homeowners along with the kickbacks to Wall St executives and help for stupid people who bought homes they can’t afford. I can only dream, right?

When the new drive arrived, I thought I would see if I could hot swap it. Smart, right? In hindsight it makes perfect sense that I would try testing out the resilency of ZFS and RAIDZ on a live array holding precious memories of my children’s births and early years when it was already starting to fail. Living life on the edge is what I like to do. Of course, when you try this on your own home file server, you might want to be a little more careful about tracing the SATA cables to make sure you actually pull the right one.

Panic. This was going to be a Friday the 13th for the ages.

There were a couple minutes there when I was literally sweating bullets. I was lucky though that the array wasn’t being used and so no data was lost. I brought c8d0 back up and cleared the zpool and all looked good.

Once I get everything up and running, I’ll write up the steps I used to make a backup, recreate the arrays, and copy my data back.

Categories: Project, Technology Tags: , ,

OpenSolaris 2008.11 Upgrade Finished

December 20th, 2008 No comments

That was pretty easy. Here’s what I did…

  • downloaded OpenSolaris 2008.11 Live CD
  • burned ISO file to a CD
  • booted fileserver from Live CD
  • double-clicked “Install OpenSolaris” on desktop of Live CD
  • answered the install questions (time zone, username, etc.)
  • ran “Add More Software” which launches the package manager
  • installed the “nas-server” collection
  • rebooted (apparently some of the nas-server config files don’t load properly until a reboot)

Then from the command line, I did some magic to get my existing 5-drive RAIDZ pool mounted on the new OS. My zpool has the unimaginative name of “storage” which you will see in the following commands.

  • zpool import -f storage
  • zpool upgrade storage
  • zfs upgrade storage

Now that my zpool was mounted and ready for the cifs service, I had to enter a few more commands to get cifs running.

  • svcadm enable -r smb/server
  • smbadm join -w
  • zfs set sharesmb=on storage
  • sharemgr show -vp (just to check)

And that was that. I just love how simple these commands are now. A lot simpler than SAMBA, but we’ll see if the new cifs service is as stable and fast.

I still had to do a little chown and chmod work to get permissions right, but then everything was perfect. The mapped network drive on my PC fired right up without any changes, and the saved “connect to server” favorite on my Mac worked immediately as well.

I have iTunes set up so that its “iTunes Music” folder is on the fileserver. So now all is well there and I can continue to manage my library from my Mac laptop, but store all the music files on the server. I feel a little better knowing that RAIDZ is protecting all those media files from bitrot, but I still need to get some backup in place for my 500+GB media folder.

Next project will be to see if I can get Firefly running to share out my movies folder to the AppleTV.

Moving to OpenSolaris

December 19th, 2008 No comments

We had a power outage last night and the interruption has created some errors on disk0. I tried to fsck, but there’s not much use unless I boot from a live CD and umount the filesystem first. I decided that maybe this was a good time to go to Solaris 10 10/08 anyways.

But then I thought, if I’m going to reinstall, why not try OpenSolaris? The new 2008.11 has some nifty features and all I really care about is getting my ZFS pool shared out via SMB/CIFS anyways. Hmmm, and doesn’t OpenSolaris have built-in CIFS support? That could be fun to try…

So 30 minutes later, I’ve got the OpenSolaris ISO downloaded and the CD has been burned. Booted into the CD, and I’m immediately greeted (well not, immediately – booting from CD is slow) by a desktop with Firefox, etc. all ready to go. This is nice.

There’s a “Install OpenSolaris” icon on the desktop. I click that and it recognizes my 6 drives. I wiped the boot drive and now it’s chugging away. It’s about 50% done after what seems like 15 minutes or so (I’m catching up on Twitter while I wait, so who knows how long it’s really been). Can’t wait to see it in action and get my zpool mounted and CIFS working.

Here’s what I’m really excited about… Apparently there are a few people that have Firefly Media Server running on OpenSolaris. This is going to be great!

Categories: Project, Technology Tags: ,

SATA Cards for Solaris 10

November 13th, 2008 No comments

My 1.25TB Solaris “experiment” is becoming more and more important as I archive my digital life and move more of my media to the AppleTV. Digital video, unsurprisingly, happens to need a lot of storage. I’ve just about got it filled up and I think I’m ready for more free space. I’ve considered alternatives like Drobo, etc. but I just don’t have a solid desktop machine to host the files besides my Solaris server. Dedicated NAS devices are still relatively expensive (consider Drobo+DroboShare, TerraStation, ReadyNas, etc. – starting around $700 for an empty chassis). It’s a lot more economical to just add more storage to my existing Solaris server. Plus, the wireless connection from my MacBook to the Solaris box has been rock solid since the last Airport update from Apple.

There are a couple ways to do this, of course. I can create a new raidz set of 3-4 drives and then expand the zpool to include this new space. I still maintain data integrity because every drive in the zpool is part of a raidz set. My other choice is to create a new raidz set of 2-4TB and then move the data from the old zpool into a new zpool. After that’s done, I would destroy the old zpool and repurpose or sell the 250GB drives. What I don’t want to do is add single drives to the zpool because then I lose data integrity if that drive fails, since it’s not part of a raidz set.

Right now, I have five 250GB drives in the existing raidz set. 4 are SATA and 1 is ATA. For my situation, I’m interested in reducing the total number of drives (to reduce both power and heat) by taking advantage of the pricing on higher density drives. At presstime, 500GB drives are around $70 ($0.14/GB), 1TB drives are $120 ($0.12/GB) and 1.5TB drives are $150 ($0.10/GB). I can do 4x 1TB drives for $480 or 3x 1.5TB drives for $450 and end up with the same usable storage (3TB).

If I wanted to save a little money, I could get 4x 750GB drives for $360 (2.25TB usable). The other drive sizes don’t make sense for trying to replace an existing 1TB array. If you had room for 4 more drives in your box, then I would consider adding a 4x 500GB or 4x 640GB set because the cost per GB is pretty close.

Of course, I’m out of SATA ports on my motherboard and almost out of drive bays in the case. If I create a new zpool and move all my data, I’ll need to have both drive sets online at least for a day while I copy 1TB of data. My plan is to get a 4x SATA II PCI card and attach the new drives there. Then I can move the zpool from the old set to the new set. Once that’s done, I’ll probably move the drives to the motherboard SATA connectors and leave the PCI card idle until I need to do something like this again.

Cheap SATA cards are plentiful, but hardware compatability with Solaris 10 is always a crapshoot. I’ve heard enough anecdotal evidence that I’m convinced I can use a card based on the Silicon Image SIL3124 chipset. In fact, the card I have in mind is this specific model. This is one of those areas that is make or break for home-brew Solaris servers. Cheap SATA drives are the whole reason for wanting to build a box like this, but sometimes finding cards with drivers that work can be a showstopper.

If anyone has other recomendations for SATA cards, I’d love to hear about them.

How to download Solaris 10 8/07

April 4th, 2008 No comments

I suppose this could be tagged “obvious” but I thought I’d warm up a bit with a simple post about where to find Solaris 10 for download. Believe it or not, you can go find the official site at and there are prominent “Get It Now!” and “downloads” links on the front page that will take you to the options for downloading Solaris or purchasing a media kit.

Registration is required to download solaris, but then you are presented with a simple page to download the DVD of Solaris x86. If you have a Windows box, there is an option to download a compressed DVD image in a self-extracting executable. This is by far the simplest method, although you do have to be patient in waiting for the download and the extraction. There is also a multi-segment download that you can use on just about any platform to build the ISO file and then burn it to a DVD.

Once on DVD, you can boot from the disc and proceed with installing Solaris 10 on your box.

Categories: Project, Technology Tags: , ,

Initial Solaris Transfer Speeds

April 7th, 2007 No comments

Using my Gigabit switch, I was able to transfer 1.25GB in 51 seconds by copying a file from my MacBook to my Solaris box. That’s 200Mbps if I convert the terms. I’ve heard of people getting close to 400Mbps on Gigabit networks, so I’m going to try and see what I can do to speed this up. In particular, I need to look into jumbo frames on both the MacBook and the Solaris box to see if there’s anything I can do to push up the speed.

I’m going to try and finish the install writeup this week. I took a lot of pix of the screen during install, but a lot of the images came out blurry (handheld P&S camera against LCD screen) I’ll try to clean up what’s usable and finish writing up the process I used.

New Features – Tags and Shared Items

April 4th, 2007 No comments

You may have noticed a couple new tabs across the top of the site. I’ve added a couple new features that I thought I would point out to my readers. I’ve been reluctant to do some of this stuff, but I figure I just need to embrace the whole “living online” thing and share a little more with the world. I’ve done two things – more prominent use of tags and more shared items from outside of the blog.

First up, Tags. You may have noticed the tags listed at the end of articles, but now I’ve added a small tag cloud to the right-hand sidebar. I’ve also added a dedicated tag page, which is available from the tabs across the top of the page. Currently the tag page shows both the full tag cloud (sorted alphabetically) and a “long tail” bar chart (sorted by frequency of use).

I’ve also done more to share items with readers and visitors. I’ve added the most recent articles from my “shared items” RSS feed to the left sidebar. This is done with Google Reader and a widget that Google provided. But really, I’ve focused on creating a new “shared” tab that highlights a number of aspects of my online life. There are a couple main features. I’ve tried to sort the page by putting the most transient information at the top and the longest lasting stuff at the bottom. So at the top, you’ll find what I’m currently reading from various RSS feeds that day. Below that is a summary of recent bookmarks I’ve posted to my account. Then we get to my personal library of books, and finally music, movies, and TV shows. The Listal portfolios don’t seem to work perfectly (they aren’t complete) but they still look cool so I’ve decided to leave the flash widgets in place rather than going with the (boring) text rolls.

I hope that you’ll find the more prominent use of tags useful and the shared items interesting. If you have other suggestions, I’d love to hear them.

Getting Started with Solaris 10 and ZFS

March 24th, 2007 11 comments

Solaris 10 LogoI’ve been itching to write more about Solaris 10 and ZFS now that I’ve got it working on my budget server. Unfortunately, I’ve been out of town this last week and haven’t had a chance to write much. Before I write up the full install and setup process, I thought I’d give you a little teaser.

You can get started at the Solaris home page. Incidentally, will redirect you to the page on, in case you forget the URL. The first thing you’ll want to do is start downloading the x86 install disc image, because this will take a while. From the Get Solaris page you need to select the download option or sign up to have a software kit sent to you in the mail. If you have a DVD drive, I would recommend the compressed DVD image download if you’re running Windows. It comes in four parts, which are heavily compressed. Once you download all four parts, the decompression and disc creation routine will take quite a while. It took about an hour (if I remember right) just to make the image on my 2.4GHz P4 machine. Add burning time on top of that.

Once you have the install disc ready to go, the rest of the process is pretty easy. You can get an overview at “How to Quickly Install the Solaris 10 1/06 OS” and there’s a pretty good Solaris x86 FAQ available from as well.

Solaris 10 is Rockin!

March 16th, 2007 5 comments

I’m proud to announce that I’ve got Solaris 10 up and running pretty smoothly on my new AMD-based system. I had some annoying compatibility issues with the first system I tried, so I switched to the ASUS M2NPV-VM motherboard with an AMD Athlon 64 X2 3600+ CPU. This board has much better support in Solaris 10 out of the box. The onboard GPU (nVidia GeForce 6150) is recognized immediately as is the onboard ethernet. So much nicer to work with…

Asus M2NPV-VM motherboard

There are some other things that I really like about this board. It’s got four USB ports and Firewire on the back. There are also headers for two more USB and one more firewire port on the motherboard. DVI out it also nice.  Check out the expansion options below…

Asus M2NPV-VM motherboard 2

Another thing that works really well for my project is that there are 2 IDE connectors and 4 SATA connectors. I plugged the boot drive and the DVD-ROM drive into the IDE port (as master/slave on one cable). Four SATA ports is the bare minimum for what I want to accomplish, but it will suffice for the next year or so. If I run out of room quicker than that, I’ll add a PCI board or something.

A couple other cool extras that come with this Asus board…

  • Firewire bracket (for the rear openings in a standard case)
  • HDTV video out bracket with component outputs (the 6150 is capable of 1080i)
  • extra IDE cables and a SATA cable

So next up is the physical build process and then Solaris installation. ZFS setup is so easy, it almost doesn’t deserve a post. I’ll see if I can do some rough benchmarks for fun though.