Recommendations for Media Server / NAS

[phanophish]

So question regarding FreeNAS 8....

Looks like ZFS provides for the fault tolerance and recovery, and I see they specifically now support thin provisioning.  What is the process for growing volumes?  Do I simply add in additional disk and the system absorbs them in to the thinly provisioned storage pool?  Or do I have to go through some process of building/incorporating them in to the existing LUNs?

Thin provisioning applies to zvols (chunks of disk space that would typically be exported via iSCSI, Fiber Channel, or FCoE). In ZFS, they are created and managed by the "zfs" command, not the "zpool" command, since they need to be created on existing storage pools.

To add space to a ZFS pool, you would install the additional drives, then add them to your pool. Since redundancy is very important, it's best to add redundant groups of drives to the pool.

I'm not sure how the FreeNAS web interface does it right now. Here's a Solaris command-line example; in this case, let's pretend we have an eight-port SAS HBA, and we're going to create a pool using two sets of mirrored drives. The HBA has already been automatically configured as controller #4 (c4) in the Solaris device addressing scheme.

Code Select Expand


zpool create mypool mirror c4t0d0 c4t1d0 mirror c4t2d0 c4t3d0


Let's say we're getting more than 2/3 full, and we want to add two more mirrored pairs:

Code Select Expand


zpool add mypool mirror c4t4d0 c4t5d0 mirror c4t6d0 c4t7d0


A different way to grow the pool, if we didn't want to use up drive slots: assuming we have the original two sets of mirrors:

BACK UP YOUR DATA FIRST

Code Select Expand


zpool set autoexpand=on mypool
zpool offline c4t0d0
(physically replace the c4t0d0 disk with a larger drive)
zpool replace -f c4t0d0
(wait for new disk to resilver)
zpool offline c4t1d0
(physically replace c4t1d0 disk with a larger drive)
zpool replace -f c4t1d0
(wait for new disk to resilver)
(optionally replace c4t2d0 and c4t3d0 in the same manner, one at a time)


As soon as the second disk in the mirror has finished resilvering, the pool will autoexpand with the additional space on the new drives.

Note that most SAS HBAs support hot swap out of the box. AHCI-compatible SATA ports can also do hot swap, but on a Solaris system, it's not enabled by default for some reason, and the line:

Code Select Expand


set sata:sata_auto_online=1


must be added to the /etc/system file (and then the system needs to be rebooted). Hot swap is wonderful for this sort of thing, as the system can keep going while you remove and replace drives.

Food for thought: If you have two hot swap drive cages, you want to arrange your mirrors so that each mirrored pair is split between the two cages. That way, if one cage goes out, the entire pool remains available via the other cage.

Always remember that RAID is not a substitute for backups.

Awesome info.  What about redundancy schemes more along the lines of RAID 5?  In my dream world I'd like a RAID5/6 array that as it fills I simply need to add additional drives, permit the parity to rebuild, and then the overall storage becomes presented as a larger massive disk.  Imagine a initial 4x2TB array running RAID 5 yielding 6TB raw.  I want to add capacity, so drop in 2 more 2TB drives, I dream of a solution that simply finds the drives and begins to expand the overall array to one that is now 10TB raw.  It allows me to reduce the % of overall storage lost to parity while also making the management simple.

Is there a place I can go that has a concise document that details all these options? 

[sparkey]

So question regarding FreeNAS 8....

Looks like ZFS provides for the fault tolerance and recovery, and I see they specifically now support thin provisioning.  What is the process for growing volumes?  Do I simply add in additional disk and the system absorbs them in to the thinly provisioned storage pool?  Or do I have to go through some process of building/incorporating them in to the existing LUNs?

Thin provisioning applies to zvols (chunks of disk space that would typically be exported via iSCSI, Fiber Channel, or FCoE). In ZFS, they are created and managed by the "zfs" command, not the "zpool" command, since they need to be created on existing storage pools.

To add space to a ZFS pool, you would install the additional drives, then add them to your pool. Since redundancy is very important, it's best to add redundant groups of drives to the pool.

I'm not sure how the FreeNAS web interface does it right now. Here's a Solaris command-line example; in this case, let's pretend we have an eight-port SAS HBA, and we're going to create a pool using two sets of mirrored drives. The HBA has already been automatically configured as controller #4 (c4) in the Solaris device addressing scheme.

Code Select Expand


zpool create mypool mirror c4t0d0 c4t1d0 mirror c4t2d0 c4t3d0


Let's say we're getting more than 2/3 full, and we want to add two more mirrored pairs:

Code Select Expand


zpool add mypool mirror c4t4d0 c4t5d0 mirror c4t6d0 c4t7d0


A different way to grow the pool, if we didn't want to use up drive slots: assuming we have the original two sets of mirrors:

BACK UP YOUR DATA FIRST

Code Select Expand


zpool set autoexpand=on mypool
zpool offline c4t0d0
(physically replace the c4t0d0 disk with a larger drive)
zpool replace -f c4t0d0
(wait for new disk to resilver)
zpool offline c4t1d0
(physically replace c4t1d0 disk with a larger drive)
zpool replace -f c4t1d0
(wait for new disk to resilver)
(optionally replace c4t2d0 and c4t3d0 in the same manner, one at a time)


As soon as the second disk in the mirror has finished resilvering, the pool will autoexpand with the additional space on the new drives.

Note that most SAS HBAs support hot swap out of the box. AHCI-compatible SATA ports can also do hot swap, but on a Solaris system, it's not enabled by default for some reason, and the line:

Code Select Expand


set sata:sata_auto_online=1


must be added to the /etc/system file (and then the system needs to be rebooted). Hot swap is wonderful for this sort of thing, as the system can keep going while you remove and replace drives.

Food for thought: If you have two hot swap drive cages, you want to arrange your mirrors so that each mirrored pair is split between the two cages. That way, if one cage goes out, the entire pool remains available via the other cage.

Always remember that RAID is not a substitute for backups.

I dream of a solution that simply finds the drives and begins to expand the overall array to one that is now 10TB raw.

Until the volume corrupts...then....ouch.

[rastasean]

Even though raid is not a replacement for backups, you may want to consider raid 10 since you're using terabytes of data and it would be faster to replicate in the event you needed to replace a drive.
I believe you can have more than one drive fail in raid 10 but its also suggested to use 4 drives of the same size.

http://support.dell.com/support/topics/global.aspx/support/entvideos/raid?c=us&l=en&s=gen



editing....
Want an off site backup but don't want the trouble of vising the site every time you need to update? DRBD could be a solution for that.
http://www.drbd.org/

mirrored networking raid 1. The nodes can be local, on different subnets, different vlans, or on a different continent (for example).

a simple setup would be having one machine at your place and dumping data onto it and the other machine at your friends/families place with it updating automatically.

In case a complete power outage takes both nodes down, DRBD will detect which of the nodes was down longer, and will do the resynchronization in the right direction.

[rjp]

What about redundancy schemes more along the lines of RAID 5?  In my dream world I'd like a RAID5/6 array that as it fills I simply need to add additional drives, permit the parity to rebuild, and then the overall storage becomes presented as a larger massive disk.  Imagine a initial 4x2TB array running RAID 5 yielding 6TB raw.  I want to add capacity, so drop in 2 more 2TB drives, I dream of a solution that simply finds the drives and begins to expand the overall array to one that is now 10TB raw.  It allows me to reduce the % of overall storage lost to parity while also making the management simple.

ZFS has a RAIDZ mode that provides one drive (RAID5), two drives (RAID6), or three drives (beyond RAID6) of redundancy. These are called RAIDZ, RAIDZ2, and RAIDZ3.

Example with four-drive RAIDZ

Code Select Expand


zpool create mypool raidz c4t0d0 c4t1d0 c4t2d0 c4t3d0


You could add another RAIDZ set later,if you have the ports available - no need to rebuild:

Code Select Expand


zpool add mypool raidz c4t4d0 c4t5d0 c4t6d0 c4t7d0


Or, if autoexpand is on, you could replace drives one at a time with bigger ones, letting each resilver, and once the last drive finishes resilvering, presto! More space!

Note that additional vdevs (drive sets, in ZFS terminology) added to an existing pool don't have to be the same configuration. You could add mirrored pairs or triplets, or add a RAIDZ2 set to an existing RAIDZ1 pool, etc.

A comprehensive guide to ZFS, though it's specific to Solaris (and OpenIndiana): http://download.oracle.com/docs/cd/E19253-01/819-5461/index.html

[phanophish]

ZFS has a RAIDZ mode that provides one drive (RAID5), two drives (RAID6), or three drives (beyond RAID6) of redundancy. These are called RAIDZ, RAIDZ2, and RAIDZ3.

Example with four-drive RAIDZ

Code Select Expand


zpool create mypool raidz c4t0d0 c4t1d0 c4t2d0 c4t3d0


You could add another RAIDZ set later,if you have the ports available - no need to rebuild:

Code Select Expand


zpool add mypool raidz c4t4d0 c4t5d0 c4t6d0 c4t7d0


Or, if autoexpand is on, you could replace drives one at a time with bigger ones, letting each resilver, and once the last drive finishes resilvering, presto! More space!

Note that additional vdevs (drive sets, in ZFS terminology) added to an existing pool don't have to be the same configuration. You could add mirrored pairs or triplets, or add a RAIDZ2 set to an existing RAIDZ1 pool, etc.

A comprehensive guide to ZFS, though it's specific to Solaris (and OpenIndiana): http://download.oracle.com/docs/cd/E19253-01/819-5461/index.html

Exactly what i was looking for.  Thanks!!!!

I work in IT so fully understand that RAID is not a backup.  I just want one simple to manage central repository for massive chunks of data.  Between my MP3 music collection, several TB of FLAC live shows, a couple more TB of RAW image files, and now a ever growing pool of HD video of the kiddo my storage needs are getting pretty ridiculous at home.  I keep a second copy on a external HD but managing the whole mess with multiple sources and targets gets to be a chore.  I'd love to build a couple of RAID systems, then simply setup RSYNCH to maintain the replication of data between the two. My critical stuff is off site either through archive.org, flickr, or youtube so while a house fire/break in might mean my source files go away, I'd at least have hi res JPEGS or HD Youtube videos of everything that really matters.   So I'm mainly looking for a nice tool to consolidate storage with easy growth and fairly robust redundancy at a reasonable cost. 

[rastasean]

Well I got freeNAS installed and running on a machine this past weekend. This was about the simplest install I've done but I'm confused about one thing and I'll talk about that in a second.

http://sourceforge.net/projects/freenas/files/
I downloaded the .7x (FreeNAS-i386-embedded-0.7.2.6694.img) embedded version since I'm installing it on a 2 gig CF card with cf to IDE adapter. Since the OS is on the CF card, I can have only 3 IDE HDDs on the motherboard I put on 1 120gig and two 80 gigs. The total space isn't that great but I knew that going into this already. Saturday night I found out the power supply fan wasn't spinning and upon taking it apart, it didn't have any bearings so I promptly replaced that as I don't want the system overheating.

Adding all three drives to the zfs pool was quite easy and just took a few clicks on the webgui, which was very fast and responsive. I turned on ftp and uploaded data at about 8-10MB a second and even created an NFS service for for linux computer. Mounted it without any issue and copied/deleted data just as fast (maybe even a little faster than ftp).

The thing I got hung up on was adding a new drive to an existing pool. I was able to format the drive to zfs and create a virtual disk but didn't figure out a way to add it to an existing pool. Anyone come across a how to on this?


So far my experience with freeNAS has been pretty good for something that's free. You can even configure fNAS to to SMART checks on the drives with a simple cronjob and email the report to you. I wouldn't say this is good enough for an entire corporate environment but for a section of a company, it could definitely be a very powerful solution. You can create users/groups; assign it to a domain controller or primary domain control (windows 2000 environment and earlier); use LDAP services, store flacs on it, etc.

[JasonSobel]

I have very little experience with FreeNAS, as I just got my server up and going yesterday. and I'm running FreeNAS 8.0 (the FreeNAS-8.0-RELEASE-i386.iso file from this link: http://sourceforge.net/projects/freenas/files/FreeNAS-8/).

That said, this is how the documentation says to add a new drive to an existing pool:

To combine two vdevs in the GUI, go to Storage -> Volumes -> Create Volume. In the Volume Name section, input the same name as an existing vdev, select the disk(s) you wish to add, choose ZFS as the filesystem, and click Add Volume.

let us know how that works for you.  I haven't yet added more drives to an existing pool, but I'm planning on doing just that several times over the next week or so.

[rastasean]

Thanks so much for this info.
I am going to re-configure the fNAS machine this week; I don't know why I didn't think of using a thumb drive for the OS instead of a CF. Using the thumb drive will free up one more chain for another IDE drive.

Good luck with your setup. What's the specs of your server you're going to have fNAS on?

[JasonSobel]

Thanks so much for this info.
I am going to re-configure the fNAS machine this week; I don't know why I didn't think of using a thumb drive for the OS instead of a CF. Using the thumb drive will free up one more chain for another IDE drive.

Good luck with your setup. What's the specs of your server you're going to have fNAS on?

I have my FreeNAS running from an 8gb usb flash drive - certainly way more space than needed to run the system, but it's what I had lying around ready to go.

For my hardware, I just put together a new computer to use as my primary machine (as discussed thoroughly in this thread: http://taperssection.com/index.php?topic=146970.0).  Now that the new computer is up and running smoothly, I repurposed my old machine to run FreeNAS.  It's a Dell Dimension 4600 that I bought in 2003.  It's got a Pentium 4 (@3gHz), and 2.5 gb RAM.  Certainly not a powerful machine by today's standards, but my goals for the FreeNAS aren't to max out performance.  Rather, I just need a central storage server for my home.  and I needed it to be as cheap as possible, in my case, free, because I installed it on hardware that I already had

[rastasean]

Your system is certainly newer than the computer I picked up. The processor on my computer is about 1.7GHz and something like one and three quarters of gigs.

I know the freenas 8, if using zfs, needs a lot of RAM. I posted a comparison between 7 and 8 a couple pages back on this thread and right now I don't have a computer I want to setup with freenas version 8 so I'll try out 7.

Good luck with your setup!

[JasonSobel]

so far so good.

FYI, I had initially created a volume, with a single 2tb drive, and filled it up halfway.
just now, I added a new 1tb drive as a new vdev to the existing zpool, following the same procedure I quoted above.

To combine two vdevs in the GUI, go to Storage -> Volumes -> Create Volume. In the Volume Name section, input the same name as an existing vdev, select the disk(s) you wish to add, choose ZFS as the filesystem, and click Add Volume.

very quick and easy, and the zpool now shows a capacity of 3tb, and it is now just 1/3 full

[JasonSobel]

Your system is certainly newer than the computer I picked up. The processor on my computer is about 1.7GHz and something like one and three quarters of gigs.

I know the freenas 8, if using zfs, needs a lot of RAM. I posted a comparison between 7 and 8 a couple pages back on this thread and right now I don't have a computer I want to setup with freenas version 8 so I'll try out 7.

Good luck with your setup!

definitely true.  with only 2.5 gigs of RAM, I've had a couple of system crashes already, and my ZFS volume never made it past 5 TB.  So I've shut it down for now, until I can get a system with a lot more RAM 

[rastasean]

yikes! The good news is you can find some on ebay usually for a good price.

My system is so old is doesn't support usb booting so I'm resorting to only three drives and so far its been pretty good. I think I just inadverently resolved an issue with a reboot...from the command line I was not able to ping, it would resolve the name but not ping.

[phanophish]

I tried FreeNAS 8 as well had the same issue with Kernel panics.  Went through most of the troubleshooting steps on these threads. 

http://sourceforge.net/apps/phpbb/freenas/viewtopic.php?f=97&t=11421
http://freebsd.1045724.n5.nabble.com/kmem-map-too-small-with-ZFS-and-8-2-RELEASE-td4029979.html

Still had the issue....  One of the threads discussed a person having the crashes with 15GB of RAM.  Seems like a pretty big flaw.....