I ran across this very cool use of ZFS and VirtualBox on blogs.sun.com. The author made a snapshot of virtualbox VM’s and then used the cloning feature to instantly create new copies of the VM for testing or other uses. All the common bits are shared by the different clones so this setup saves gigs of disk storage. Very clever.
Last week, I replaced a drive in my RAIDZ that was starting to report errors. To be clear, I was actually getting errors on 2 or 3 drives, but most of them were on c7d0. When I was trying to see if I could hotswap out c7d0, I unplugged c8d0 and almost crapped my pants. When I realized how loose the SATA cables were in getting c8d0 back up and running in the array, I decided I better do this with the power off.
Here are the steps I took to make a backup and then rebuild my array. I decided to build the arrays a little differently, so these steps aren’t needed if you are just replacing a drive. “zpool replace storage c7d0″ with a new drive connected to controller 7 would have sufficed in my case.
To make a backup, I plugged in a 1TB USB drive and used fdisk to clear it into one big partition. I used the device driver manager in the Gnome GUI to tell me that it was sitting on c9t0d0. I created a ZFS filesystem named “backup” on it and then used “zpool send storage | zfs receive -f backup” to copy the filesystem to the other drive. This worked beautifully and only 12 hours or so later, I had a complete copy of all 900GB of data.
This points out the futility of backup for these large arrays. Even if you do have the drives, tape, or whatever, it’s hard to get more than 1 or 1.5GB per minute. That’s only 60GB to 90GB per hour. It takes a long time to backup a TB of data.
Once the backup was done, I used “zpool destroy -f storage” to break the array. I did this because I wanted to rebuild it as one 4 drive array with SATA drives and a single drive array with the other IDE drive in it. This way when I eventually upgrade to 1.5TB drives, I can just use four instead of the five drives that are in there now. With the drives ready to go, I created the new array and filesystem with the command “zpool create raidz storage c5d0 c6d0 c7d0 c8d0″. I also created a “movies” zpool for DVD rips that didn’t have to be redundant with the command “zpool create movies c4d0″.
Now that I had the two new arrays ready, I just used “cp” to copy the data back to the new arrays and put the folders in the right places (rips to /movies and everything important to /storage- the raidz array).
And no, this isn’t a comment on my server’s self-esteem. My RAIDZ pool is reporting that it’s got problems. It figures it happens on Friday the 13th. Weird, huh?
Actually I first saw the warnings a couple weeks ago when a scrub returned several errors in the zpool. The data was still good though because of checksumming in ZFS. I was getting a little anxious because I keep my iTunes library on the server along with raw footage from home movies. Important stuff to our family that I wouldn’t want to tell my wife that I had lost.
Most of the errors were on C7D0 (one of the four SATA drives in my setup) so I spent a week or two debating about whether I wanted to just upgrade the four 250GB drives to 1TB or 1.5TB drives. In the end, I decided on hoarding cash and I ordered a 250GB drive for $40. I’d still like to upgrade the array to 1.5TB drives when they hit $99 each, but I can wait to see if this new stimulus package will include free drives for homeowners along with the kickbacks to Wall St executives and help for stupid people who bought homes they can’t afford. I can only dream, right?
When the new drive arrived, I thought I would see if I could hot swap it. Smart, right? In hindsight it makes perfect sense that I would try testing out the resilency of ZFS and RAIDZ on a live array holding precious memories of my children’s births and early years when it was already starting to fail. Living life on the edge is what I like to do. Of course, when you try this on your own home file server, you might want to be a little more careful about tracing the SATA cables to make sure you actually pull the right one.
Panic. This was going to be a Friday the 13th for the ages.
There were a couple minutes there when I was literally sweating bullets. I was lucky though that the array wasn’t being used and so no data was lost. I brought c8d0 back up and cleared the zpool and all looked good.
Once I get everything up and running, I’ll write up the steps I used to make a backup, recreate the arrays, and copy my data back.
That was pretty easy. Here’s what I did…
- downloaded OpenSolaris 2008.11 Live CD
- burned ISO file to a CD
- booted fileserver from Live CD
- double-clicked “Install OpenSolaris” on desktop of Live CD
- answered the install questions (time zone, username, etc.)
- ran “Add More Software” which launches the package manager
- installed the “nas-server” collection
- rebooted (apparently some of the nas-server config files don’t load properly until a reboot)
Then from the command line, I did some magic to get my existing 5-drive RAIDZ pool mounted on the new OS. My zpool has the unimaginative name of “storage” which you will see in the following commands.
- zpool import -f storage
- zpool upgrade storage
- zfs upgrade storage
Now that my zpool was mounted and ready for the cifs service, I had to enter a few more commands to get cifs running.
- svcadm enable -r smb/server
- smbadm join -w
- zfs set sharesmb=on storage
- sharemgr show -vp (just to check)
And that was that. I just love how simple these commands are now. A lot simpler than SAMBA, but we’ll see if the new cifs service is as stable and fast.
I still had to do a little chown and chmod work to get permissions right, but then everything was perfect. The mapped network drive on my PC fired right up without any changes, and the saved “connect to server” favorite on my Mac worked immediately as well.
I have iTunes set up so that its “iTunes Music” folder is on the fileserver. So now all is well there and I can continue to manage my library from my Mac laptop, but store all the music files on the server. I feel a little better knowing that RAIDZ is protecting all those media files from bitrot, but I still need to get some backup in place for my 500+GB media folder.
Next project will be to see if I can get Firefly running to share out my movies folder to the AppleTV.
We had a power outage last night and the interruption has created some errors on disk0. I tried to fsck, but there’s not much use unless I boot from a live CD and umount the filesystem first. I decided that maybe this was a good time to go to Solaris 10 10/08 anyways.
But then I thought, if I’m going to reinstall, why not try OpenSolaris? The new 2008.11 has some nifty features and all I really care about is getting my ZFS pool shared out via SMB/CIFS anyways. Hmmm, and doesn’t OpenSolaris have built-in CIFS support? That could be fun to try…
So 30 minutes later, I’ve got the OpenSolaris ISO downloaded and the CD has been burned. Booted into the CD, and I’m immediately greeted (well not, immediately – booting from CD is slow) by a desktop with Firefox, etc. all ready to go. This is nice.
There’s a “Install OpenSolaris” icon on the desktop. I click that and it recognizes my 6 drives. I wiped the boot drive and now it’s chugging away. It’s about 50% done after what seems like 15 minutes or so (I’m catching up on Twitter while I wait, so who knows how long it’s really been). Can’t wait to see it in action and get my zpool mounted and CIFS working.
Here’s what I’m really excited about… Apparently there are a few people that have Firefly Media Server running on OpenSolaris. This is going to be great!
My 1.25TB Solaris “experiment” is becoming more and more important as I archive my digital life and move more of my media to the AppleTV. Digital video, unsurprisingly, happens to need a lot of storage. I’ve just about got it filled up and I think I’m ready for more free space. I’ve considered alternatives like Drobo, etc. but I just don’t have a solid desktop machine to host the files besides my Solaris server. Dedicated NAS devices are still relatively expensive (consider Drobo+DroboShare, TerraStation, ReadyNas, etc. – starting around $700 for an empty chassis). It’s a lot more economical to just add more storage to my existing Solaris server. Plus, the wireless connection from my MacBook to the Solaris box has been rock solid since the last Airport update from Apple.
There are a couple ways to do this, of course. I can create a new raidz set of 3-4 drives and then expand the zpool to include this new space. I still maintain data integrity because every drive in the zpool is part of a raidz set. My other choice is to create a new raidz set of 2-4TB and then move the data from the old zpool into a new zpool. After that’s done, I would destroy the old zpool and repurpose or sell the 250GB drives. What I don’t want to do is add single drives to the zpool because then I lose data integrity if that drive fails, since it’s not part of a raidz set.
Right now, I have five 250GB drives in the existing raidz set. 4 are SATA and 1 is ATA. For my situation, I’m interested in reducing the total number of drives (to reduce both power and heat) by taking advantage of the pricing on higher density drives. At presstime, 500GB drives are around $70 ($0.14/GB), 1TB drives are $120 ($0.12/GB) and 1.5TB drives are $150 ($0.10/GB). I can do 4x 1TB drives for $480 or 3x 1.5TB drives for $450 and end up with the same usable storage (3TB).
If I wanted to save a little money, I could get 4x 750GB drives for $360 (2.25TB usable). The other drive sizes don’t make sense for trying to replace an existing 1TB array. If you had room for 4 more drives in your box, then I would consider adding a 4x 500GB or 4x 640GB set because the cost per GB is pretty close.
Of course, I’m out of SATA ports on my motherboard and almost out of drive bays in the case. If I create a new zpool and move all my data, I’ll need to have both drive sets online at least for a day while I copy 1TB of data. My plan is to get a 4x SATA II PCI card and attach the new drives there. Then I can move the zpool from the old set to the new set. Once that’s done, I’ll probably move the drives to the motherboard SATA connectors and leave the PCI card idle until I need to do something like this again.
Cheap SATA cards are plentiful, but hardware compatability with Solaris 10 is always a crapshoot. I’ve heard enough anecdotal evidence that I’m convinced I can use a card based on the Silicon Image SIL3124 chipset. In fact, the card I have in mind is this specific model. This is one of those areas that is make or break for home-brew Solaris servers. Cheap SATA drives are the whole reason for wanting to build a box like this, but sometimes finding cards with drivers that work can be a showstopper.
If anyone has other recomendations for SATA cards, I’d love to hear about them.
I suppose this could be tagged “obvious” but I thought I’d warm up a bit with a simple post about where to find Solaris 10 for download. Believe it or not, you can go find the official site at www.solaris.com and there are prominent “Get It Now!” and “downloads” links on the front page that will take you to the options for downloading Solaris or purchasing a media kit.
Registration is required to download solaris, but then you are presented with a simple page to download the DVD of Solaris x86. If you have a Windows box, there is an option to download a compressed DVD image in a self-extracting executable. This is by far the simplest method, although you do have to be patient in waiting for the download and the extraction. There is also a multi-segment download that you can use on just about any platform to build the ISO file and then burn it to a DVD.
Once on DVD, you can boot from the disc and proceed with installing Solaris 10 on your box.
I finally got around to upgrading my Solaris box to 8/07 (Update 4). I guess the most significant comment I could make is that it was almost entirely uneventful. I had to check my SAMBA config because the services run slightly different in this update. Still very easy and quick.
I chose to download the single compressed DVD image again (only available when downloading to a Windows box) and then burned the DVD from there. It takes a while to download and then a while more to decompress the DVD image, but everything worked on the first try. I took my server down to boot from the DVD and run the upgrade install process and everything was updated in short order.
I upgraded my ZFS storage pool with “zpool upgrade” and that was done in few moments. I love the succinctness of ZFS commands.
One of the new features I’m excited about is “zpool status -v” so that I can more easily get a list of files that are showing errors. I’m away for the weekend, but I’ll check this more thoroughly when I get back home.
The other thing I need to check is to see if the “disappearing cursor” bug is fixed in this update. Boy, that one is annoying when I switch into the GUI desktop.
For those of you that don’t get all fired up about ZFS, you may be excited to find out more about the new containers functionality and the support for running Linux binaries in a container.
I was pretty intrigued by the short bit Scoble wrote about Drobo today after Gizmodo posted their story. Data Robotics has come up with an almost idiot-proof external storage array. To hit the highlights, it connects to a Mac or PC with USB 2.0 (480 Mbps theoretical maximum) and requires no special software – Drobo manages the storage all by itself – so it just looks like a big USB drive to your computer.
There are a few features that I really appreciate about their solution…
- There are simple LED indicator lights that tell you if the drives are healthy and if the array is getting full.
- SATA drives are added through the front of a very slick tool-less chassis. The drive chassis may be the coolest feature I’ve seen – I’d like to have that setup in my server case.
- Drobo makes sure that your data is protected.
- You can expand the size of the array at any time by adding additional drives (there are four slots) or swapping out smaller drivers for larger drives.
- It supports both Mac and PC hosts
- The external case has a cool design that I wouldn’t mind seeing on my desk.
It might not be ideal for everyone. It’s still an external drive enclosure so it requires a host computer and cannot be connected to ethernet directly as Network Attached Storage (NAS). I’m not sure how quiet it is either, which would be important for a HTPC media storage application. That said, it could be a great solution for the photographer or graphic artist that needs a large storage solution that can protect her data and still grow over time.
If you want to see what I’m talking about, I highly recommend the demonstration video on the Drobo site.
About the only thing that I don’t like about Drobo is the grammatical error in their tagline, “Drobo – whose minding the storage.”
For those looking for a NAS, this newish solution from Promise looks interesting. Of course, I like my personal budget server (based on Solaris), but it doesn’t have the convenience of these stand-alone solutions, even if it does cost a lot less and have more flexibility and capabilities.
Using my Gigabit switch, I was able to transfer 1.25GB in 51 seconds by copying a file from my MacBook to my Solaris box. That’s 200Mbps if I convert the terms. I’ve heard of people getting close to 400Mbps on Gigabit networks, so I’m going to try and see what I can do to speed this up. In particular, I need to look into jumbo frames on both the MacBook and the Solaris box to see if there’s anything I can do to push up the speed.
I’m going to try and finish the install writeup this week. I took a lot of pix of the screen during install, but a lot of the images came out blurry (handheld P&S camera against LCD screen) I’ll try to clean up what’s usable and finish writing up the process I used.