Taperssection.com
Gear / Technical Help => Post-Processing, Computer / Streaming / Internet Devices & Related Activity => Topic started by: Brian Skalinder on May 25, 2006, 11:23:49 PM
-
Please note I now recommend FreeFileSync. (http://sourceforge.net/projects/freefilesync/)
Originally posted this in another thread (http://taperssection.com/index.php?topic=65087.msg876215#msg876215), but thought it deserved it's own thread:
Just a thought for some folks not wanting to mess around with RAID, or wanting to save some money: pick up multiple HDDs and do your own automated (non-real time) redundancy handling. For example:
- Get two drives (internal or external, or one of each); designate one drive X and the other drive Y (or whatever letters you wish)
- Store all your data on drive X and use drive Y as your redundant copy
- Install and configure Free FileSync (http://sourceforge.net/projects/freefilesync/) to mirror the entire drive X to drive Y. Use a Task Scheduler for scheduling (I do mine nightly using Windows 7 Task Scheduler).
- Once set up and configured, the initial run of Free FileSync (http://sourceforge.net/projects/freefilesync/) will copy all of drive X's contents to drive Y.
- Future runs (which users may schedule) will then only copy new files, update those that have changed, and delete the backups if the original is deleted. (Alternatively, you may customize the sync rules for Free FileSync (http://sourceforge.net/projects/freefilesync/) to follow, depending on your own requirements.)
While not real-time, the Free FileSync. (http://sourceforge.net/projects/freefilesync/) solution will provide redundancy protection against (daily / hourly / whatever) losses, depending on how frequently you perform the scheduled sync. I figure I can recover from any one day / half day / 4 hour (whatever) losses easily enough. It's the loss of many days (or weeks, months) worth of data that scares me.
Note: I say "redundancy" and not "backup" because storing data on multiple HDs is not really an effective backup strategy. For proper backup, it's best to have at least one regulary-updated off-site copy.
I've tried and rejected the following, based on my experiences:
- Argentum. If the source drive (or directory) disappears, Argentum deletes the target drive, assuming the user intentionally deleted it and wants the synced target drive / directory deleted, too. So if your source HDD fails, Argentum deletes your backup. Dumb!
- DeltaCopy. Generally works okay, but I've had consistent permissions issues with the synced directories & files.
- SyncToy. I know some people for whom it works very well. But for me: sometimes copies all the files it should, and sometimes doesn't. Inconsistent = b-a-d.
-
MD5 checksums can also be used to tell if a file is still good or the copy of it is good.
Gordon
-
Good advise, but you can save yourself $25 with a little simple scripting.
On Windoes make a batch file (a text file with a .bat extension) with something like this in it:
xcopy d:\*.* e: /s /d
Then setup the command scheduler to run that at whatever interval you like.
Also, the new MS utility SyncToy for WinXP will do the same thing with a nice gui interface and a little smarter logic that will amke it faster and more efficent.
SyncToy can be downloaded from here:
http://www.microsoft.com/downloads/details.aspx?FamilyId=E0FC1154-C975-4814-9649-CCE41AF06EB7&displaylang=en
The help file shows you how to schedule it. I use it to back up my wife's laptop to my server (on a RAID5 volume) every night.
-
Good advise, but you can save yourself $25 with a little simple scripting.
On Windoes make a batch file (a text file with a .bat extension) with something like this in it:
xcopy d:\*.* e: /s /d
Then setup the command scheduler to run that at whatever interval you like.
Great tip, Matt! I should've thought of it. And now, due to a fatal flaw in Argentum Backup (see original post), I'm using a similar batch file:
set mydate=%date:~10%-%date:~4,2%-%date:~7,2%
set mytime=%time:~0,2%-%time:~3,2%
xcopy i: j: /d /s /c /f /y > f:\z_backup\log\I-J_%mydate%_%mytime%.log
set mydate=%date:~10%-%date:~4,2%-%date:~7,2% | captures the date for use in my log filename
set mytime=%time:~0,2%-%time:~3,2% | captures the time for use in my log filename
i: | my source drive
j: | my destination drive
/d | copies any new or updated files from source to destination
/s | copies all subdirectories and files
/c | continues copying through errors (not positive I want to do this, yet)
/f | displays full source and destination paths/filenames I capture in the log
/y | suppress prompting to confirm an overwrite of an existing destination file
> | sends the results of the operations to a log file
f:\z_backup\log\i-j_%mydate%_%mytime%.log | path and filename of the log file
This option will only copy new or updates files from source to destination, and does not deletions at all.
Can't use SyncToy or Robocopy (referenced in another thread) since I'm on Win2K and have no desire to change.
-
That's a bummer.. it all comes back to xcopy.. gotta love the 80's! ;)
I like rsync but it isn't windows native. And of course rsync handles this case and can either delete upon change or not.
It is so hard to keep up with backups and off-siting..
-
I used abakt before but not for media. I just looked and unfortunately the author stopped development but it seemed to work ok for me when I did use it. Which wasn't very extensively but I did see it has command line function as well so a little tweaking and I'm sure it could be ran on a schedule.
it is free also fwiw...
-
That's a bummer.. it all comes back to xcopy.. gotta love the 80's! ;)
I like rsync but it isn't windows native. And of course rsync handles this case and can either delete upon change or not.
It is so hard to keep up with backups and off-siting..
i'm using a windows build of rsync called DeltaCopy (http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp).
-
i'm using a windows build of rsync called DeltaCopy (http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp).
Ah, excellent...I'll give DeltaCopy a try. Thanks!
-
I emailed AB support about this problem and they basically said: it's not a bug, it's by design, and changing it would be really difficult. What a lame response.
that is just laughable. sorry about your troubles brian. I'm reconfiguring my media server this weekend. I upgraded to 2 500gb hard drives and I don't want to mess with raid so I'm going to do basically the same thing you've done.
-
I'd do the same... but 2 500gig drives wouldn't be enough :P
+T for making me think about this again.
yeah I'm afraid that 500gb is only going to last me about 6 more months. 24 bit recording eats up hard drive space quick especially If I save off unedited 4 channel stuff.
-
i'm up over 5TB and i'm running out of room!
-
Teddy, whom I have not seen on the board for some time now, recommended that if the data is not in three separate places it is not safe. This is a good, prudent approach, especially if someone is paying you. I plan on getting a USB 500GB for music, having it on the PC on an internal 500GB and burning a copy to DVD of every stage as it develops. "I'm sorry, I lost it," does not work with customers.
-
^^^ Don't leave us hanging. What happened? Are you in a taper protection program?? ;o)
-
Another nice program to help automate these processes is called SyncBack. Can be simple, can be complex...very configurable. I use it to backup photos to a RAID server that I run, but it can be used just as easily to mirror a drive to another.
-
Here's a free virtual Network Attached Storage (NAS) solution that runs on a FreeBSD platform http://www.freenas.org/ (http://www.freenas.org/) - a lot of people are using this to support the SlimServer (Squeezebox) software. I've been trying to find a large volume *secure* storage solution as well.
There's also a variety of appliance based solutions to choose from; Drobo, ReadyNAS, Windows Home Server (due this month), Buffalo Terastation, etc. - most of these are SATA drive enclosures that have a RAID similar protection (either OS or hardware RAID).
-
Good advise, but you can save yourself $25 with a little simple scripting.
On Windoes make a batch file (a text file with a .bat extension) with something like this in it:
xcopy d:\*.* e: /s /d
Then setup the command scheduler to run that at whatever interval you like.
Great tip, Matt! I should've thought of it. And now, due to a fatal flaw in Argentum Backup (see original post), I'm using a similar batch file:
set mydate=%date:~10%-%date:~4,2%-%date:~7,2%
set mytime=%time:~0,2%-%time:~3,2%
xcopy i: j: /d /s /c /f /y > f:\z_backup\log\I-J_%mydate%_%mytime%.log
set mydate=%date:~10%-%date:~4,2%-%date:~7,2% | captures the date for use in my log filename
set mytime=%time:~0,2%-%time:~3,2% | captures the time for use in my log filename
i: | my source drive
j: | my destination drive
/d | copies any new or updated files from source to destination
/s | copies all subdirectories and files
/c | continues copying through errors (not positive I want to do this, yet)
/f | displays full source and destination paths/filenames I capture in the log
/y | suppress prompting to confirm an overwrite of an existing destination file
> | sends the results of the operations to a log file
f:\z_backup\log\i-j_%mydate%_%mytime%.log | path and filename of the log file
This option will only copy new or updates files from source to destination, and does not deletions at all.
Can't use SyncToy or Robocopy (referenced in another thread) since I'm on Win2K and have no desire to change.
I have been using xcopy for about 4 years now. It is SOoo easy (as you can see by Brian's script) I did not know about the log file though so I just added that feature. Now if someone could help me with a command line FTP script that would be worth a few T's to me.
-
I'd strongly recommend an rsync based approach over xcopy. Here's why:
rsync uses checksums to verify that data has been copied correctly. You can easily have rsync verify that a copy is correct. It will recalc the checksum on every source file and every destination file. I don't think xcopy uses checksums?
-
I'd strongly recommend an rsync based approach over xcopy. Here's why:
rsync uses checksums to verify that data has been copied correctly. You can easily have rsync verify that a copy is correct. It will recalc the checksum on every source file and every destination file. I don't think xcopy uses checksums?
I will give DeltaCopy a whirl then. Thanks for the info. I love MD5's and verification tools. They are our friends.
-
Ah, excellent...I'll give DeltaCopy a try. Thanks!
Working great, after I enabled Protected Storage service to fix a scheduling problem. Thanks again. :)
-
Ah, excellent...I'll give DeltaCopy a try. Thanks!
Working great, after I enabled Protected Storage service to fix a scheduling problem. Thanks again. :)
excellent. glad to hear it. i've been using it for a while now, and it definitely does the trick. i backup 1 drive a night on my PC.
-
with all respect, unless someone is hardware illiterate, a RAID0 card (as cheap as`$20, top quality under $60) is a better alternative, because it eliminates the requirement to manually back up.
the biggest failure point of a backup system is human intervention. the possibility of making a mistake and copying the wrong drive over, or simply forgetting to back up is very real.
if you can install a hard drive, you can install a RAID card
Argentum costs money and your computer has to be on all the time to take advantage of scheduled backups
Originally posted this in another thread (http://taperssection.com/index.php?topic=65087.msg876215#msg876215), but thought it deserved it's own thread:
Just a thought for some folks not wanting to mess around with RAID, or wanting to save some money: pick up multiple discs and do your own automated (non-real time) redundancy handling. For example:
- Get two 250 GB USB/Firewire/internal drives; designate one drive X and the other drive Y
- Store all your data on drive X (use drive Y as your backup)
Install an app like Argentum's Backup (http://www.argentuma.com/backup.html) ($25) to automate duplicating the data from drive X to drive Y on some fixed schedule (weekly, daily, hourly, whatever suits your purposes)
After an initial copy of all of drive A's data to drive B, the app will perform only incremental backups, i.e. the app will only copy new or changed data; I bet there're free apps available, too, or you could even write your own batch file easily enough, but FWIW I like Argentum's Backup because it's cheap and has the option to store data in non-proprietary formats (unlike many s/w backup apps)
START EDIT
I cannot recommend Argentum Backup any longer and strongly recommend anyone using it STOP, and find another solution (I'm now using an xcopy script, see my latest reply to the thread). I'm kicking myself for not finding this fatal flaw earlier (a result of no real testing of my backup routines until just now...dumb dumb dumb). Argentum Backup's fatal flaw: if the source is not available for any reason - network goes down, drive goes offline / crashes for any reason, etc. - Argentum assumes the user intentionally deleted / removed the source and will update the destination (backup) accordingly, i.e. it will delete the destination (backup)! WTF?!? Example:
- I have all my data stored on my F: drive.
- I want to backup my data from F: onto my G: drive.
- I set up Argentum Backup (AB) to backup my data from F: to G: nightly.
- AB runs the first time, and copies all data from F: to G:.
- The next day, my F: drive suffers a catastrophic failure.
- AB runs again.
- AB deletes all the files on G:.
I emailed AB support about this problem and they basically said: it's not a bug, it's by design, and changing it would be really difficult. What a lame response.
END EDIT
Not real-time, but it will provide redundancy protection against (daily / hourly / whatever) losses, depending on how frequently you perform the automated backup. I figure I can recover from any one day / hour's losses easily enough. It's the loss of many days (weeks, months) worth of data that scares me - I have no desire to re-transfer 300 GB of data from CD-R/DVD-R to HD.
Note 1: I say "redundancy" and not "backup" because storing data on multiple HDs is not really an effective backup strategy
Note 2: RAID or the above scenario is NOT a sufficient backup program. A comprehensive backup plan should also include physical backups on optical (CD-R/DVD-R) or magnetic (tape) media and off-site backups.
-
with all respect, unless someone is hardware illiterate, a RAID0 card
I assume you meant RAID1.
When people who do storage for a living and are considered experts in the field say they won't trust consumer raid solutiatons, I listen to'em..
The above, combined with two unfortunate personal experiences with cheap RAID cards, convinced me to bail on cheapo RAID solutions. Not going back to the (IMO) false sense of security from cheapo RAID cards, thanks, and think others should be aware of the pitfalls and alternative options.
the biggest failure point of a backup system is human intervention. the possibility of making a mistake and copying the wrong drive over, or simply forgetting to back up is very real.
Once scheduled, there's no human intervention required for DeltaCopy SyncToy Free FileSync.
Argentum costs money and your computer has to be on all the time to take advantage of scheduled backups
If you read the thread, I've edited to recommend -against- using Argentum (now recommend DeltaCopy SyncToy Free FileSync). You're right that using these tools requires the PC to be on. :shrugs:
As for it costing money, so do the cheapo RAID cards. I'm willing to pay a few bucks for good software that does what I want it to do (unfortunately, Argentum doesn't, and fortunately, DeltaCopy SyncToy Free FileSync does and is free): CDWave, for example, which it seems very few around here actually purchase, is a great, inexpensive option for tracking -- well worth purchasing a license, even though free alternatives exist.
-
yeah, it was like 4am, i obviously meant raid1.
one other tidbit about consumer-level cards and raid recovery.
raid1 data is almost always written in a standard format, where in event of controller failure, one of the volumes could be easily read by a standard ide/sata controller. not so for raid0, which has a variety of block and stripe sizes generally specific to the controller.