Topic: Back-up question

[yousef]

I find myself in the position of having many, many hard drives with back-ups of varying vintage spread across them. I know I have at least two copies of everything spread between them but would really like to streamline and better organize how everything is kept.

Is there a program out there that will catalogue the contents of each drive, identify the most recent version of each file and facilitate a complete backup of everything in a single location?

Started doing this process manually a few weeks ago - reams of paper, coloured stickers on each drive, masses of confusion and stress...

[livingdna]

This is what I use: http://www.2brightsparks.com/syncback/syncback-hub.html. When running a backup it will first identify which files differ between your source and backup and then allow you to only write those files to your backup.

Be sure to set up your backup profiles correct so that you won't overwrite your source files instead of your backup files and it's a very powerful tool for easy backup.

Oh and it's free

[yousef]

Sounds promising...

I'll give it a try and see if it works for me. Many thanks 

[Gordon]

I've been using sync back for a couple years and recently started having issues.  I found this

http://www.goodsync.com/

and like it a lot better.  very user friendly with lots of options.

[Scooter123]

I just bought a pair of NAS Buffalo RAID JBODs for back up.  And use Folder Clone for the software. 

Folder Clone compares the original drive (source) and copies that data over to the back up (target) and makes the contents of the two drives identical. 

When you change the source, say by changing a single file, Folder clone only changes that single discreet file in the target, so back ups can be quite.  Unlike other programs which back up the whole drive fresh each time, or encrypt the backup with a proprietary file structure or file name, Folder Clone literally clones the source drive.  Its about $30

[Brian Skalinder]

Is there a program out there that will catalogue the contents of each drive, identify the most recent version of each file and facilitate a complete backup of everything in a single location?

I don't know of one that will catalogue the contents of multiple HDDs, compare, and facilitate copying only the most recent version of a file from across multiple drives drives onto a single backup drive.  Looks like the others mentioned above are similar to the s/w I use (FreeFileSync), and will compare 2 HDDs, identify the differences, and allow copying of more recent files from drive A to drive B.  Not sure of the others, though I assume they do similar, but FFS also gives you the option to:

• Choose how to determine if files are different:  by file time and size, or by content (binary comparison)
• Define filters to exclude certain files, file types, or folders
• Prior to sync, view the file comparison results and defined sync actions for each file, and per file change the sync action, if desired (e.g. copy from left to right, right to left, do nothing, etc.)
• Establish custom sync options based on the comparison result, e.g. what action to take if the file:  exists in left but not right, exists in right but not left, is newer in left than right, is newer in right than left, has a conflict or cannot be categorized.

Sounds like you're on  the right track in consolidating your backups.  Whenever I run out of space on my backup HDDs, I get new, larger HDDs precisely so I have don't have to worry about multiple variations or versions of backups spanning multiple HDDs.  (Though at one point I did have more than one backup HDDs with various content across each; I found two manageable, but probably wouldn't want to mess around with more, unless I absolutely had to do so...e.g., if I couldn't fit all my data onto a single HDD, in which case I'd probably opt to build a big-ass array as I know some here have done.)

If there isn't a tool that does so automatically, sounds like you'll need to backup 'HDD A' onto 'HDD Backup'.  Then backup 'HDD B' onto 'HDD Backup', ensuring that <a> only files that don't yet exist on 'HDD Backup' are copied from 'B' to 'Backup', and <b> for files that do exist on 'HDD 'Backup', only copy those that are newer on 'B' than on 'Backup'.  And so on for each HDD.

Whatever software you end up using, if you have the backups scheduled and automated, test what happens when the source drive goes offline.  At least one backup program I know of (not mentioned here intentionally) assumes the absence of the source drive means it was deleted, and therefore updates the backup to match...by deleting the backup files.  (Brilliant, eh?)

[yousef]

Thank you for the further replies.

After looking into a handful of options, I've resigned myself to doing the initial work (the consolidation of multiple backups) manually. It looks like I've got lots of options to keep things up to date once I've managed to make sure that I have a full set of files in one location.

I think the mistake I've made here has been trying to economize on hard drives: if I'd gone bigger with each purchase rather than trying to buy smaller drives and spread the back-up across them I don't think I'd be this mess. That and many hurried dumps of video and audio files to anywhere there was space on the day before a gig...

Whatever software you end up using, if you have the backups scheduled and automated, test what happens when the source drive goes offline.  At least one backup program I know of (not mentioned here intentionally) assumes the absence of the source drive means it was deleted, and therefore updates the backup to match...by deleting the backup files.  (Brilliant, eh?)

This sounds quite terrifying. Thanks for the warning.

[stevetoney]

I have mixed emotions about auto backups.  The bottom line is that I don't use it, but I've given it thought in the past. 

On the one hand, even though I do periodic manual backups, I have been caught with my pants down by a hard drive failure where I hadn't backed some recent stuff up (ya know...I'll do it next weekend type of thing).  Obviously, auto backup would have saved me. 

On the other hand, I've had fears of issues such as mentioned in the previous posts, where the backup doesn't go off like you thought it was gonna and all of a sudden your computer has done something automatically that you never wanted it to do.  After reading the above, I'll probably just keep doing manual backups.  I don't process all that much data on my home machines on a weekly basis anyway, so frequent auto backups are probably less important than if alot of files were changed say on a daily basis.

[rastasean]

On unix, and presuming you've labeled everything well, just use tree.


Here's a basic example:

/var/backups/
├── alternatives.tar.0
├── alternatives.tar.1.gz
├── apt.extended_states.0
├── apt.extended_states.1.gz
├── apt.extended_states.2.gz
├── dpkg.diversions.0
├── dpkg.diversions.1.gz
├── dpkg.statoverride.0
├── dpkg.statoverride.1.gz
├── dpkg.status.0
├── dpkg.status.1.gz
├── dpkg.status.2.gz
├── dpkg.status.3.gz
├── group.bak
├── gshadow.bak
├── passwd.bak
└── shadow.bak

0 directories, 17 files

[bombdiggity]

I think the mistake I've made here has been trying to economize on hard drives: if I'd gone bigger with each purchase rather than trying to buy smaller drives and spread the back-up across them I don't think I'd be this mess.

There's something to be said for not having all your eggs in one basket.  Small drives means less can be lost if one catches fire... 

The alternative way to view it is to have the original small ones become "backups" that are archived away (disconnected and boxed in a safe place) while the large drives everything gets dumped to become the working drives.  You can back up the large ones where things are changing on a regular basis to other large ones.   Redundancy seems the key to a happy life... 

[adrianf74]

I run a HTPC in my household and spent forever trying to find a solution that would allow me to keep some level of redundancy in my media collection.  For years, I've kept at least three copies of ANY data that I wouldn't want to lose.  Photos, master audio files, personal documents, etc.  Today, I keep two sets at home plus a third drive that has critical data backed up in a third location.

I'm in the PC world and use a program called Disparity which, essentially, allows me to have redundancy with my main HTPC by using one drive as a parity drive (similar in fashion to PAR2 files with RAR sets).  In my HTPC, I've got 20TB + 4TB (parity) and that allows either the parity drive to fail outright (and a new set gets created when a new drive is added) _OR_ one of the other drives (with data) to fail and then that "missing" data is recreated between the data on the other drives plus the parity set.  I use Hard Disk Sentinel Pro to monitor the S.M.A.R.T health status of my drives around the clock and it emails me if a drive temperature gets too high or if a drive's health fails by 1%.  It will shut the system down if the drive's health drops below a specific point (I've got 80% listed as a threshold).

I then have two USB 3.0 portable drives (one 2TB and one 1TB drive) which have copies of family photos, master audio files, documents, etc. that I use GoodSync with to confirm all files match what's on the main drive in the HTPC.  This make me feel pretty secure in that my files are safe.   I've seen too many people lose important data.   

I know I could've just gone the mirrored drive route but for 20TBs of drives, we're talking a major footprint. 

[anhisr]

I have an NAS that holds 5 HD using Synology RAID system.  One of the HD is a hot spare that the NAS will move everything too is a drive fails a SMART test.  That is what is my usable drives at home.  Then I keep two 4TB hard drives at my parents that i keep a copt of the NAS.