Topic: I'm considering a RAID stack, ISO tech help

[travelinbeat]

So I'm not 100%computer inept, but when it comes to this sort of thing, I'm no wizard.  What I would like to do is get five 1TB harddrives and build a RAID configuration that would allow me 4TB of storage while simultaneously allowing for the death of one drive without the loss of any data.  From what I've read, this is a RAID 10 or RAID 01 or something...  I don't really know.  I know it's pretty popular though (I'm sure if you're in this thread you know what I'm talking about).  It stripes data in fragments across the four remaining disks while the one disk keeps the primary file (I think).  Anyways, I have a Toshiba laptop, probably about 5 years old now, and I've already maxed out a couple of externals (one 750 and one 350) and now I'd love to make the leap into something a little bigger and a little more secure.  I would like to know if I could buy just any random internal harddrive, and any random 5-bay external enclosure for this configuration, or if there is some special care that must be paid to assure compatibility, etc.  Furthermore, I would like to know once I get the drives and the enclosure, if there is anything that would prevent my computer from recognizing or reading/writing to/from the new drive(s) (I would like to partition them to be read as one drive, as I understand this is what most people do with RAID configurations).  Anyways, please lt me know if you have any advice or insights, I certainly appreciate any and all help at this early stage!

my computer: http://tinyurl.com/37y37yx

[fleish]

I'm afraid the data you've put forth doesn't jive. RAID 0+1 & 1+0 require an event number of disks. Unless you mean for the 5th disk to serve as a spare, you can't make a RAID 01/10 array out of 5 disks. Assuming you would use 4 disks in the array and keep the 5th as a spare, you wouldn't get 4TB of usable space using either. You could build a RAID5 array out of the 5 disks and you'd get ~4TB of space which would survive the loss of a drive. There's a performance decrease for writes to RAID5 so depending on what you are going to use it for that may or may not be an issue. I'd recommend reading http://en.wikipedia.org/wiki/Nested_RAID_levels as a primer (there's a link in there to nested/hybrid RAID sets which is what RAID 01/10 are). You can also use a RAID calculator, such as http://www.ibeast.com/content/tools/RaidCalc/RaidCalc.asp to see how much space you'd get with different configurations. I'm assuming you are planning to do this is software vs. buying a hardware RAID controller. If so as long as your software supports the RAID configuration there shouldn't be anything keeping it from accessing the RAID array upon creation.

[travelinbeat]

I'm afraid the data you've put forth doesn't jive.

Thanks for the clarification-- I'll definitely read the link.  The main purpose of the RAID array would be storage, but I would like to be able to read off of it with ease.  Ideally, I will drop media (huge 24/96 wavs, etc) onto my harddrive (the cpu's internal harddrive), process them, then archive them on the external RAID.  I don't mind if it takes a while to copy the data over there, but I would mind if the processing from RAID > CPU would be too slow / weak to allow me to listen to those files back or watch movies off of the RAID or something.  I do not have any plans of using the RAID as a base of a 2nd OS, nor a platform off of which to launch programs, or anything like that.  In doing some further topical research, I think that I was thinking of RAID 5 when I was saying RAID 10, so please excuse that miscommunication.

I also see you address the issue of a RAID controller-- this is another issue that I am quite unable to wrap my head around.  I am 99% sure that my computer does not have any built in RAID controller, and when I search for them on Newegg.com, I get such an outrageous margin of cost that it makes me quite wary about doing anything ($30 - $500+???).  I don't know if they make external controllers or even perhaps controllers built into enclosures.  Do they make RAID controllers for laptops?  If so, what's a good one and how can I tell if it will fit / work?  Also, perhaps to nullify most of my more recent questions, your post seems to indicate the possibility of a RAID controller not even being necessary-- did I understand that correctly?  If so, would this software solution entail opening up some silly GUI everytime I wanted to deal with the array or would it be some super-hidden background thing that would just turn on with every bootup and allow me to use the array as just another drive?

Thanks for all of the help sorry for all of the questions!

[it-goes-to-eleven]

Fwiw...

All of my friends who are experts in computer storage, without exception, recommend not using RAID for home systems.. And they avoid doing so.

Raid is not for backups, it is for keeping your system up and running when a drive fails.

There are a lot of gotchas that can happen with raid that could actually cause you to lose your data.  Especially if you don't have a lot of experience with it professionally.

[travelinbeat]

Fwiw...

All of my friends who are experts in computer storage, without exception, recommend not using RAID for home systems.. And they avoid doing so.

Raid is not for backups, it is for keeping your system up and running when a drive fails.

There are a lot of gotchas that can happen with raid that could actually cause you to lose your data.  Especially if you don't have a lot of experience with it professionally.

So what's the better alternative?

[OFOTD]

Couple of questions first.

1.  What equipment (computers, drives, OS) do you currently own?   
2.  What is your current budget to get up and running with a storage solution?
3.  How much time do you want to dedicate to installation and maintenance?

[travelinbeat]

1.  What equipment (computers, drives, OS) do you currently own?   

I posted a link to my computer's website at Toshiba.  I run XP.  The computer is stock and I'm running a pair of externals to bump up my storage capability-- one is a 500GB Seagate internal in an external case, the other is some cute little Western Digital thing that doesn't require power (just the USB) and is smaller than a standard harddrive.  I think this one is 300GB.

2.  What is your current budget to get up and running with a storage solution?

Actual liquid the budget is very narrow.  I would like to buy things in steps-- maybe an enclosure first (and maybe buy only an enclosure first if I get something like one of the cool $300 Rosewells), then a couple drives, then maybe a couple more, etc).  This would allow me to do this a lot more easily.

3.  How much time do you want to dedicate to installation and maintenance?

Not quite sure how to answer this... I don't mind taking a day or two to set it up (or a day or two every so often once I expand the number of drives I have).  Similarly, I wouldn't mind if I needed to do things like run defrags periodically overnight, etc.  I would rather not have to actually tend to this storage solution, by which I mean I am looking for this to alleviate burdens, not add to them.  I hope that that makes sense.

[OFOTD]

Let me preface by saying that without knowing what your computer knowledge level is these suggestions are what I would give someone with average computer experience.

With the laptop you have I see two different options for you.

1. Go with a NAS box that will accept new drives.  Something like this:  http://www.newegg.com/Product/Product.aspx?Item=N82E16822122010 

2. Turn an old desktop into a home server and add a drive/RAID box like this:  http://www.newegg.com/Product/Product.aspx?Item=N82E16816111139

Option #1 is $300 upfront + the cost of drives.   You'd be limited to four drives total so limited future expandability but it is one small little network box.

Option #2 is probably $400 upfront + the cost of drives. You'd probably be able to get 10-12 drives depending on the size of the old desktop.   Unlimited expandability depending on old desktop.

If you are on a super tight budget i'd recommend starting with an old desktop and buying Windows Home Server for $100.   Add an external enclosure box as you get more money then add more drives.

So for $100 more than the small NAS box you could have a true home server that did more than just basic storage.  Things like nightly backups, MCE additions, seedbox, etc. 

[travelinbeat]

Thanks OFO, good input.  Where would a RAID controller factor in to option 2, and what's a good one?  And if I set up option two, how can I access that desktop true my laptop?

[OFOTD]

Thanks OFO, good input.  Where would a RAID controller factor in to option 2, and what's a good one?  And if I set up option two, how can I access that desktop true my laptop?

Just to give a quick example of my setup.   I have a desktop that I have installed Windows Home Server on.  $100 for an OEM copy from Newegg.   Plugged into the desktop via an included RAID card that came with enclosure: http://www.newegg.com/Product/Product.aspx?Item=N82E16816111141   I got it for $300 a while back.     Inside that enclosure I have it filled up with drives.

Windows Home Server (WHS) uses a pseudo-software RAID that does what you are looking to do.  Add XX drives, if one fails then you have some redundant backup on other drives.    So for example I have say 10 folders on my WHS enclosure.  Movies, music, FLAC, pictures, etc.   I only have my family pictures and personal files set to duplicate.   If I lose a FLAC I know I can get it on the net.  Same goes with movies.   Easy to replace.  My family pictures not so much.  So in that sense this let's you maximize your storage space.   It also will backup all of our laptops and computers to the storage enclosure at times we specify.  Another thing to note is that you can add external USB/FW drives to your WHS machine and use those drives in your storage pool as well.

I stream FLAC's, movies, recorded tv shows, music and all sorts of things via my WHS.   Mine is using wired as well as wireless connections without problem.   In your situation with a laptop I would assume you'd do this wireless.    Oh and it sets up a web link directly to your server so you can access it's contents from anywhere.

The system requirements for WHS are pretty low so if you have a desktop not being used or know someone with an older one then you have a prime candidate for WHS.  Also WHS is made to run headless (no monitor, keyboard or mouse).  Set it up once and stick it out of the way.   It requires less that 15 minutes of maintenance a month all from the remote interface.

Hope some of that makes sense.

Here is a snapshot from last month with my original enclosure attached with 8 1.5TB drives in it as well as a 500GB internal in the desktop i'm using.



So you can see each drive and it's status all from the WHS interface.   I have added a second enclosure since this screen grab. Same unit but 8 2TB drives instead.  I'm running almost 29TB of storage space now.

[travelinbeat]

I'm running almost 29TB of storage space now.

Storage space is bigger in Texas!!!

Well that all sounds great-- it would allow expandability with drives of different brands and models, it would allow me to access the files from the other side of the planet, and it would be a really somewhat reasonably cheap way to accomplish all of what I want.  I could store the hardware in another room of the house on a shelf or something and just link up on a wireless network.  The main expenses to get off the ground are the $350 for the enclosure and a few hundred for a couple of internal 1TB's.  On that note, I've noticed that some HD's are SATA and some are some other format or something.... but I don't understand anything about that.  Also-- do I still need to set pins on harddrives or am I aging myself by asking such a question (ie: "master" vs. "slave")?

Thanks again!

[sparkey]

Just to give a quick example of my setup.   I have a desktop that I have installed Windows Home Server on.  $100 for an OEM copy from Newegg.   Plugged into the desktop via an included RAID card that came with enclosure: http://www.newegg.com/Product/Product.aspx?Item=N82E16816111141   I got it for $300 a while back.     Inside that enclosure I have it filled up with drives.

Windows Home Server (WHS) uses a pseudo-software RAID that does what you are looking to do. 

Just speculation, but it sounds like an OS crash on your machine with the RAID adapter and outboard array would be much more forgiving than OS corruption of the Windows file system providing RAID functionality.   If the Windows file system doing software RAID corrupts, I'd say your logical array would go down with it in flames...the outboard (physical) array probably would still be able to recognize the content on the enclosure after a nuke and pave on the Windows OS.  Does that make sense?

Josh

[Scooter123]

I love RAID.  This is my second computer with it, and I have two arrays, one for the hard drive and one for the back up.

Two types of RAID, hardware and software.  Go with the hardware.  This would require a card either taking up a slot, or some advanced mother boards have a RAID card installed.  I've sinc'ed four (4) SATA drives each 2tb with RAID 0 giving me a drive with 8tb on it.  I have a small fast drive for the O/S. 

Those drives are all backed up via eSATA to a cheap 4 bay SATA array also RAIDED which requires another card. 

[mattmiller]

Also-- do I still need to set pins on harddrives or am I aging myself by asking such a question (ie: "master" vs. "slave")?

No jumpers to set on SATA hard drives.  Just feed them power and a data connection to the motherboard and they're good to go.  Unless you get DOA drives, which I just got two of a couple of weeks ago.

[Brian Skalinder]

FWIW, my take on RAID:

To RAID or not for me depends on <1> my budget, and <2> whether it's critical to have real-time redundant data.  IME, inexpensive / consumer-level RAID controllers aren't terribly reliable, which means I'd have to budget more than I'd like for a HQ RAID card.  And personally, my critical data doesn't change frequently enough to warrant (real-time) RAID, so implementing RAID would be overkill.  So instead, I employ reasonable redundancy without RAID, in my case "mirrored" on a nightly basis.  I'll never lose enough data on a daily basis that it will cause undue harm or inconvenience.  Others may define a different period between mirroring, like 12 hrs, 8 hrs, 4 hrs, etc., depending on individual needs.

In addition, I've had unpleasant experiences with RAID based on s/w or inexpensive RAID cards, so I don't bother anymore since I'm not willing to cough up the cash for a HQ RAID controller.  IMO, RAID makes most sense when one needs real-time redundancy, which most of us don't.

I think it's worth considering reasonable redundancy without RAID.

[rjp]

On the contrary, I prefer software RAID (such as MD-RAID in Linux or, better yet, ZFS in Solaris or FreeBSD) to hardware RAID cards or cheap "fakeraid" cards. If your hardware RAID controller goes kablooey, you will most likely need to get an exact replacement (or, if you're slightly luckier, at least one from the same manufacturer).

For building a ZFS-based appliance without having to use command lines, FreeNAS (FreeBSD-based), or NexentaStor (Solaris-based) are good places to start. Note that the free version of NexentaStor limits you to 12 TB, though.

Always remember, though, RAID is not a replacement for backups.

[Shadow_7]

Do you even need raid?  Hardware raid is to increase your read and write times, by spreading the physical I/O over several relatively slow devices.  Software raid gives you the ability to combine several drives into one big virtual drive.  And then you add redundancy on top of all that to degrade your performance, but increase your ability to lose part of the hardware without losing part of the data.  Unless you're running a business or web server do you really either of those functionalities?

I have a usb docking station and a number of SATA 1TB-ish drives.  Greater than 4TB of storage now, which is on par with my last employers capacity for the entire company(1,000+ employees).  What I do is keep stuff on their own drives and any that are hotter than their peers, or noisier, that is fairly noticeable on a docking station.  Normally those questionable drives have other external marks that might indicate that they shouldn't have passed QA in the first place.  What I do is use one drive for unedited video, one drive for unedited audio, one drive for projects, any active projects are probably on the computer I'm using at the time.  But all originals reside on multiple places.  If I have a spare drive I use it for backups of important stuff.

Bear in mind that HDDs are spinning platters so if they're on, they're spinning.  Mechanically speaking moving parts have friction, friction causes parts to wear and get smaller, ultimately resulting in failure.  With the docking station route, most of my drives remain OFF most of the time.  With a raid array most of your drives will be ON most of the time.  Do the math.  That plus powering several drives increases your electrical bill, in addition to the heat those drives produce and the A/C to cool the room. 

It really depends on how you want your data and access to it configured.  90% or more of my data is not stuff that I access daily or even need access to most of the time.  So why subject it to potential failures and direct access by malicious applications.  Just buy a bunch of those bamboo bathroom trays and put some drawer matting on the bottom.  Two drives per tray and stack them up on a book shelf.

[sparkey]

On the contrary, I prefer software RAID (such as MD-RAID in Linux or, better yet, ZFS in Solaris or FreeBSD) to hardware RAID cards or cheap "fakeraid" cards. If your hardware RAID controller goes kablooey, you will most likely need to get an exact replacement (or, if you're slightly luckier, at least one from the same manufacturer).

How would you rebuild your raid stack in the event of OS corruption?

[phanophish]

Good article on Windows OS based software RAID...

http://www.tomshardware.com/reviews/windowsxp-make-raid-5-happen,925.html

The skinny is it works pretty well and is portable between systems.  Probably best to not use it for a boot drive but for data storage it would be ideal.

[rjp]

How would you rebuild your raid stack in the event of OS corruption?

(note: I'm an old hand with UNIX-type systems and command lines...)

In my case, I'm running OpenSolaris on a mirrored pair of drives. If one side of the mirror goes bad, I can pull out the drive (it's in a hot-swap cage), chuck in a new drive, put a Solaris partition on it, and tell the OS to "replace" the old device. I'd also have to make sure to install GRUB (the boot loader) on the new drive, but that's a single command.

If *both* drives went bad, I'd have to boot OpenSolaris from a live USB stick (no CD-ROMS in my storage box) and restore the root pool (the OS installation) from backups. Yes, I keep backups.

My main storage pool is separate from the OS, with 8x500 GB drives arranged as a RAIDZ2 (two drives can fail without data loss). All of those are in hot-swap cages as well, and the storage pool works on entire drives (no partitioning needed) and the autoreplace property is set. If a drive fails in that pool, I can just swap in a new one on-the-fly with no command line intervention. The ZFS RAID-Z system can support one to three drives worth of redundancy (a RAIDZ3 can take three drive failures without data loss). There are tradeoffs, of course; more redundant drives means less capacity, and very large RAIDZ3 arrays (greater than 8-10 drives) start to run into performance issues. ZFS arrays can be built from groups of mirrors or groups of RAID-Z arrays, and if you started out with just a mirrored pair, you can readily add additional mirrored pairs to increase storage capacity.

ZFS also has a snapshot capability; you can quickly take a snapshot of a file system, and then roll back future changes if something went bad (or fetch an old version of a file from the snapshot). If you are sharing a ZFS data set over CIFS (Windows file sharing), Windows can access previous versions of files or folders (right-click the file or folder, select Properties, click the "Previous Versions" tab). Snapshots don't occupy extra disk space unless files are modified, and only the modified disk blocks occupy extra space.

In ZFS, all disk blocks are checksummed, and all reads verify the checksums; it doesn't trust the drives to maintain data integrity. Since I've actually experienced a drive that silently corrupted data (fortunately, I didn't lose anything), this is very important to me. For extra paranoia, I use ECC RAM - all recent AMD chipsets support ECC, and the Asus boards I use enable it in the BIOS.

Note that FreeNAS and NexentaStor are small enough to run from USB sticks, disk-on-module devices, CF cards, etc. They also have web-based interfaces to manage the system, so they'd be easier for most people to use. They are "appliances" rather than full OS installations...

One disadvantage of OpenSolaris in the current time frame: After Oracle took over Sun, they wasted no time in shutting off further development of OpenSolaris in favor of commercial Solaris (and quit releasing binary snapshots of OpenSolaris). Things are a bit in flux right now, but the Illumos Project is now working on a fork of the final public release of OpenSolaris source code, and the OpenIndiana Project has just released a beta distribution currently based on the final OpenSolaris code, with plans to move to Illumos in the near future. (To analogize with Linux, Illumos is the kernel and support utilities, OpenIndiana is a distribution that has desktop and server software, and so on.)