I ran across this very cool use of ZFS and VirtualBox on blogs.sun.com. The author made a snapshot of virtualbox VM’s and then used the cloning feature to instantly create new copies of the VM for testing or other uses. All the common bits are shared by the different clones so this setup saves gigs of disk storage. Very clever.
Last week, I replaced a drive in my RAIDZ that was starting to report errors. To be clear, I was actually getting errors on 2 or 3 drives, but most of them were on c7d0. When I was trying to see if I could hotswap out c7d0, I unplugged c8d0 and almost crapped my pants. When I realized how loose the SATA cables were in getting c8d0 back up and running in the array, I decided I better do this with the power off.
Here are the steps I took to make a backup and then rebuild my array. I decided to build the arrays a little differently, so these steps aren’t needed if you are just replacing a drive. “zpool replace storage c7d0″ with a new drive connected to controller 7 would have sufficed in my case.
To make a backup, I plugged in a 1TB USB drive and used fdisk to clear it into one big partition. I used the device driver manager in the Gnome GUI to tell me that it was sitting on c9t0d0. I created a ZFS filesystem named “backup” on it and then used “zpool send storage | zfs receive -f backup” to copy the filesystem to the other drive. This worked beautifully and only 12 hours or so later, I had a complete copy of all 900GB of data.
This points out the futility of backup for these large arrays. Even if you do have the drives, tape, or whatever, it’s hard to get more than 1 or 1.5GB per minute. That’s only 60GB to 90GB per hour. It takes a long time to backup a TB of data.
Once the backup was done, I used “zpool destroy -f storage” to break the array. I did this because I wanted to rebuild it as one 4 drive array with SATA drives and a single drive array with the other IDE drive in it. This way when I eventually upgrade to 1.5TB drives, I can just use four instead of the five drives that are in there now. With the drives ready to go, I created the new array and filesystem with the command “zpool create raidz storage c5d0 c6d0 c7d0 c8d0″. I also created a “movies” zpool for DVD rips that didn’t have to be redundant with the command “zpool create movies c4d0″.
Now that I had the two new arrays ready, I just used “cp” to copy the data back to the new arrays and put the folders in the right places (rips to /movies and everything important to /storage- the raidz array).
And no, this isn’t a comment on my server’s self-esteem. My RAIDZ pool is reporting that it’s got problems. It figures it happens on Friday the 13th. Weird, huh?
Actually I first saw the warnings a couple weeks ago when a scrub returned several errors in the zpool. The data was still good though because of checksumming in ZFS. I was getting a little anxious because I keep my iTunes library on the server along with raw footage from home movies. Important stuff to our family that I wouldn’t want to tell my wife that I had lost.
Most of the errors were on C7D0 (one of the four SATA drives in my setup) so I spent a week or two debating about whether I wanted to just upgrade the four 250GB drives to 1TB or 1.5TB drives. In the end, I decided on hoarding cash and I ordered a 250GB drive for $40. I’d still like to upgrade the array to 1.5TB drives when they hit $99 each, but I can wait to see if this new stimulus package will include free drives for homeowners along with the kickbacks to Wall St executives and help for stupid people who bought homes they can’t afford. I can only dream, right?
When the new drive arrived, I thought I would see if I could hot swap it. Smart, right? In hindsight it makes perfect sense that I would try testing out the resilency of ZFS and RAIDZ on a live array holding precious memories of my children’s births and early years when it was already starting to fail. Living life on the edge is what I like to do. Of course, when you try this on your own home file server, you might want to be a little more careful about tracing the SATA cables to make sure you actually pull the right one.
Panic. This was going to be a Friday the 13th for the ages.
There were a couple minutes there when I was literally sweating bullets. I was lucky though that the array wasn’t being used and so no data was lost. I brought c8d0 back up and cleared the zpool and all looked good.
Once I get everything up and running, I’ll write up the steps I used to make a backup, recreate the arrays, and copy my data back.
My 1.25TB Solaris “experiment” is becoming more and more important as I archive my digital life and move more of my media to the AppleTV. Digital video, unsurprisingly, happens to need a lot of storage. I’ve just about got it filled up and I think I’m ready for more free space. I’ve considered alternatives like Drobo, etc. but I just don’t have a solid desktop machine to host the files besides my Solaris server. Dedicated NAS devices are still relatively expensive (consider Drobo+DroboShare, TerraStation, ReadyNas, etc. – starting around $700 for an empty chassis). It’s a lot more economical to just add more storage to my existing Solaris server. Plus, the wireless connection from my MacBook to the Solaris box has been rock solid since the last Airport update from Apple.
There are a couple ways to do this, of course. I can create a new raidz set of 3-4 drives and then expand the zpool to include this new space. I still maintain data integrity because every drive in the zpool is part of a raidz set. My other choice is to create a new raidz set of 2-4TB and then move the data from the old zpool into a new zpool. After that’s done, I would destroy the old zpool and repurpose or sell the 250GB drives. What I don’t want to do is add single drives to the zpool because then I lose data integrity if that drive fails, since it’s not part of a raidz set.
Right now, I have five 250GB drives in the existing raidz set. 4 are SATA and 1 is ATA. For my situation, I’m interested in reducing the total number of drives (to reduce both power and heat) by taking advantage of the pricing on higher density drives. At presstime, 500GB drives are around $70 ($0.14/GB), 1TB drives are $120 ($0.12/GB) and 1.5TB drives are $150 ($0.10/GB). I can do 4x 1TB drives for $480 or 3x 1.5TB drives for $450 and end up with the same usable storage (3TB).
If I wanted to save a little money, I could get 4x 750GB drives for $360 (2.25TB usable). The other drive sizes don’t make sense for trying to replace an existing 1TB array. If you had room for 4 more drives in your box, then I would consider adding a 4x 500GB or 4x 640GB set because the cost per GB is pretty close.
Of course, I’m out of SATA ports on my motherboard and almost out of drive bays in the case. If I create a new zpool and move all my data, I’ll need to have both drive sets online at least for a day while I copy 1TB of data. My plan is to get a 4x SATA II PCI card and attach the new drives there. Then I can move the zpool from the old set to the new set. Once that’s done, I’ll probably move the drives to the motherboard SATA connectors and leave the PCI card idle until I need to do something like this again.
Cheap SATA cards are plentiful, but hardware compatability with Solaris 10 is always a crapshoot. I’ve heard enough anecdotal evidence that I’m convinced I can use a card based on the Silicon Image SIL3124 chipset. In fact, the card I have in mind is this specific model. This is one of those areas that is make or break for home-brew Solaris servers. Cheap SATA drives are the whole reason for wanting to build a box like this, but sometimes finding cards with drivers that work can be a showstopper.
If anyone has other recomendations for SATA cards, I’d love to hear about them.
I suppose this could be tagged “obvious” but I thought I’d warm up a bit with a simple post about where to find Solaris 10 for download. Believe it or not, you can go find the official site at www.solaris.com and there are prominent “Get It Now!” and “downloads” links on the front page that will take you to the options for downloading Solaris or purchasing a media kit.
Registration is required to download solaris, but then you are presented with a simple page to download the DVD of Solaris x86. If you have a Windows box, there is an option to download a compressed DVD image in a self-extracting executable. This is by far the simplest method, although you do have to be patient in waiting for the download and the extraction. There is also a multi-segment download that you can use on just about any platform to build the ISO file and then burn it to a DVD.
Once on DVD, you can boot from the disc and proceed with installing Solaris 10 on your box.
I finally got around to upgrading my Solaris box to 8/07 (Update 4). I guess the most significant comment I could make is that it was almost entirely uneventful. I had to check my SAMBA config because the services run slightly different in this update. Still very easy and quick.
I chose to download the single compressed DVD image again (only available when downloading to a Windows box) and then burned the DVD from there. It takes a while to download and then a while more to decompress the DVD image, but everything worked on the first try. I took my server down to boot from the DVD and run the upgrade install process and everything was updated in short order.
I upgraded my ZFS storage pool with “zpool upgrade” and that was done in few moments. I love the succinctness of ZFS commands.
One of the new features I’m excited about is “zpool status -v” so that I can more easily get a list of files that are showing errors. I’m away for the weekend, but I’ll check this more thoroughly when I get back home.
The other thing I need to check is to see if the “disappearing cursor” bug is fixed in this update. Boy, that one is annoying when I switch into the GUI desktop.
For those of you that don’t get all fired up about ZFS, you may be excited to find out more about the new containers functionality and the support for running Linux binaries in a container.
With all the buzz in the air this week about ZFS, I knew I had to get blogging again. Unfortunately, starting up again after taking a few weeks off to get through finals is proving harder than I thought it would be. I needed something to get me jumpstarted again and this week’s coverage of ZFS has been the kick in the pants that I needed.
I’ve been itching to write more about Solaris 10 and ZFS now that I’ve got it working on my budget server. Unfortunately, I’ve been out of town this last week and haven’t had a chance to write much. Before I write up the full install and setup process, I thought I’d give you a little teaser.
You can get started at the Solaris home page. Incidentally, www.solaris.com will redirect you to the page on sun.com, in case you forget the URL. The first thing you’ll want to do is start downloading the x86 install disc image, because this will take a while. From the Get Solaris page you need to select the download option or sign up to have a software kit sent to you in the mail. If you have a DVD drive, I would recommend the compressed DVD image download if you’re running Windows. It comes in four parts, which are heavily compressed. Once you download all four parts, the decompression and disc creation routine will take quite a while. It took about an hour (if I remember right) just to make the image on my 2.4GHz P4 machine. Add burning time on top of that.
Once you have the install disc ready to go, the rest of the process is pretty easy. You can get an overview at “How to Quickly Install the Solaris 10 1/06 OS” and there’s a pretty good Solaris x86 FAQ available from sun.drydog.com as well.
I’m proud to announce that I’ve got Solaris 10 up and running pretty smoothly on my new AMD-based system. I had some annoying compatibility issues with the first system I tried, so I switched to the ASUS M2NPV-VM motherboard with an AMD Athlon 64 X2 3600+ CPU. This board has much better support in Solaris 10 out of the box. The onboard GPU (nVidia GeForce 6150) is recognized immediately as is the onboard ethernet. So much nicer to work with…
There are some other things that I really like about this board. It’s got four USB ports and Firewire on the back. There are also headers for two more USB and one more firewire port on the motherboard. DVI out it also nice. Check out the expansion options below…
Another thing that works really well for my project is that there are 2 IDE connectors and 4 SATA connectors. I plugged the boot drive and the DVD-ROM drive into the IDE port (as master/slave on one cable). Four SATA ports is the bare minimum for what I want to accomplish, but it will suffice for the next year or so. If I run out of room quicker than that, I’ll add a PCI board or something.
A couple other cool extras that come with this Asus board…
- Firewire bracket (for the rear openings in a standard case)
- HDTV video out bracket with component outputs (the 6150 is capable of 1080i)
- extra IDE cables and a SATA cable
So next up is the physical build process and then Solaris installation. ZFS setup is so easy, it almost doesn’t deserve a post. I’ll see if I can do some rough benchmarks for fun though.
The new parts are on order now. I ran into some issues with the previous gear, particularly with the Intel integrated graphics adapter and the onboard LAN. I decided to go with an AMD setup because it’s been reported to work with Solaris 10. I chose the Asus M2NPV-VM motherboard, which even has an entry on the official HCL, even if there are scant details available there. I expect the video and the LAN to be recognized on install so I’ll be disappointed if that doesn’t work out.
The one departure I took was to save $20 and go with GeIL memory rather than the Corsair XMS sticks I was looking at. I’ve been buying Corsair for a couple of years now, but the good reviews on newegg.com convinced me to try the GeIL. The ones I picked come with racing orange heat spreaders, so they have to be fast, right? It’s still DDR2 800 (PC2-6400) RAM and should run at 4-4-4-12 if I push the voltage to 2.1. One of the things I like about Asus boards is the easy tweaking for things like this.
The main reason that I went with this setup was to really try and push the cheapest build possible to get close to 1TB of raw storage. The M2NPV-VM has 2 IDE ports and 4 SATA ports. I’ve got four 250GB SATA drives for storage and a 120GB IDE drive for the boot volume. I’ve also got a spare 250GB IDE drive (in an external enclosure at the moment) so I could potentially hook up 1.25TB of raw storage. That should yield 1TB in a RAIDZ pool (with single parity). So here’s where I am with the gear right now…
- $84 – Asus M2NPV-VM motherboard
- $95 – AMD Athlon 64 X2 3600+
- -$10 – combo discount on CPU/MB
- $84 -GeIL 2x512MB DDR2 800 RAM
- $70 – Antec SmartPower 2.0 SP-500 ATX12V 500W Power Supply
- -$30 – rebate on PSU
- $260 – 4x250GB SATA drives
- $15 – shipping
- Free – spare tower case, PS2 keyboard, mouse, SATA cables, Arctic Silver, IDE boot drive, etc.
So $568 total (assuming I can still drive a calculator) for 1000GB of storage. Maybe 1.2TB if I throw in the other drive (I already have a 500GB external – do I really need another one?). I would’ve been closer to $500 if I’d gone with something like the Celeron D or the Athlon 64 (non X2) and a slightly cheaper motherboard (w/ slower, cheaper RAM) but I figured the extra dough will save me some hassle when I’m all through with this. Considering that’s only $100 more than the external 1TB drives, I think I’m doing pretty well.
I think I’ll post some pics this time when I build the server. I always like seeing other people’s builds so I thought I could share mine with the world. In particular, I want to show some pics of the power supply (I know, I know – super geeky) because I really like the modular cable design on this unit.