Topic: Archive / backup practices?

[Brian Skalinder]

We probably need to come up with some good practice backup techniques for all of this since a lot more people are moving to non-tape solutions for recording. Then we need to put those in the FAQ section.

We've discussed this in bits and pieces, but good idea to have a formal round of it and archive the info, Wayne.  I'll draw from these posts the most common / effective techniques people are using for archiving and backup.

IMO, one needs at least 3 copies of a recording to sufficiently minimize the risk of losing a recording due to archiving problems - damaged media, failed hard drive, misplaced disc, etc.  I think I have a pretty good plan in place - here's what I do:

[1]  Master WAV backup
I archive the un-touched master WAV from my recording, on a data disc (DVD or CD).  Archiving as WAV data instead of FLAC makes it somewhat more likely I'm able to recover more of the data in the event of a media problem (e.g. scratched disc).  Some may ask:  Why archive as WAV data instead of FLAC?  Here's why.

[2]  Finalized WAV backup
After tracking, applying (non-destructive) fades, completing an info file, noting in a file for myself any editing¹ I've done aside from normal tracking and fades (rare), I archive the finalized recording as data onto to CD/DVD.

[3]  On-site Finalized FLAC backup
Same files as [2], except I've FLAC'd the WAV files.  Stored on mirrored hard drives (RAID1 / automated non-RAID redundancy), so if one drive bites it, I can slap in another drive and rebuild the array.  I use these files for primary listening:  PC > playback system.

[4]  Off-site Finalized FLAC backup
The above are all well and good, but what if my house burns down, or is flooded, or my DVD binders and PC are stolen, or the dogs whiz on my DVD binders and both HDs in the same day?  I seed or host almost all of my recordings via BitTorrent or FTP, so I know I always have my recording archived off-site with fellow tapers and music lovers.  If ever I needed to rebuild my collection of master recordings, I could probably get 90% of them back this way.  The other 10% I haven't shared for one reason or another (usually b/c the band or recording stinks so bad I wouldn't ever listen to it), and I'm not terribly concerned with recovering them.  Recovering the 90% may take take some time, but I could do it if necessary.

¹ If performing substantial editing (EQ, compression, etc.) - something I do only very, very rarely - archive more than a single copy of the unedited master [1], by applying [2-3] to the unedited master as well - and maybe even [4].

Verify backups
Verify your media backups (CD/DVD) are written properly at the time of writing - if they're flawed for some reason, and you don't catch it, the backups won't do you much good, will they?  Do it manually, or automatically (I use Nero's "verify" function), or both.

Spot-check
Periodically spot-check your archived media - pull out an old CD/DVD backup and try to restore from it.  Doing so may catch the start of a broad-based batch media failure.  A single media failure may indicate a problem with other media of the same type / manufacture.  If you encounter a problem with a spot-check, immediately:

• replace the corrupted backup disc with a new, uncorrupted copy
• check discs of similar age and/or manufacture to determine if it's an isolated incident or represents a trend
• ensure your redundant backups are still in good shape

Self-discipline and diligence
Exercise the self-discipline and diligence to perform ALL backup steps for EVERY recording.  This is easier than it sounds.  The one (or two, or ten) times you don't, and you lose a recording, you'll kick yourself in the arse repeatedly.  If it's worth doing, it's worth doing right.

Anyone have alternative methods, or critiques of the method outlined above?

[MattD]

One relatively easy way to achieve [4] is to set up a computer elsewhere (parents', sibling's, significant other's, or good friend's house) and run it as an FTP server. When you're done with your recording, upload it to the offsite machine.

There's also archive.org! If you have a show by a band that allows uploads, there is absolutely no reason not to put your show up there. It's free backup and an easy way to share.

[kskreider]

Or just get a external that is the same size as your hdd and set up a X Copy DOS script that backs up every night via the task scheduler.  I do this at work on EVERYONE's workstation and they don't even know.  I have everything backing up to the server and then the server backing up to may workstation where it copies to a external hdd which I carry out.  I also rip to DVD's as a removable media.   

If you leave for an extended vacation, etc take out the external and drop it off at a friend or family's place.

Always best to have back-ups of your backup's back-up. 

All it took was one horror story and I have not, nor ever will, lose any data...ever. 

[sml42]

[3]  On-site Finalized FLAC backup
Same files as [2], except I've FLAC'd the WAV files.  Stored on mirrored hard drives (RAID1), so if one drive bites it, I can slap in another drive and rebuild the array.  I use these files for primary listening:  PC > playback system.

I've heard horror stories of things going badly wrong with friends' RAID arrays. Enough that my personal preference would be to use optical media / external HDD / offsite for backup instead.

Equally, I've heard lots of success stories with RAID. If you decide to use RAID, it goes without saying you should test thoroughly! Are the drives hot-pluggable? Pull one out, pop in a new one and make sure you can actually recover the data!

(this is akin to good sysadminning where you practice recovery on a test system - pull the power, see if you can recover; trash your partition table, see if you can recover, delete your root partition and see if you can recover, deliberately reformat your discs... you get the idea, basically try to break your system in the most imaginative way possible and make sure you can recover. that way when your *real* system goes down you know *exactly* what to do )

best regards,
stephen

[zowie]

You don't have to go through all that complexity for #4.  Just do two of #1 and/or #2 and keep a set of the discs off site. What's the big deal?

[Nick Graham]

This thread is particularly interesting to me, because I discovered several master FLAC discs of mine were fubar last month, and I didn't use a proper backup plan in advance. At the time I was using an Edirol R4, so no DAT backup either. Both discs appeared to be an example of my burner not liking Memorex DVD+R, but it was enough to scare me into changing my habits.

I now burn the untracked/unfaded wav + untracked/unfaded FLAC file onto a DVD+R (Taiyo Yuden), the complete "mastered" FLAC files onto a Mitsui CDR, and finally copy the final FLAC files again onto an external HDD. I've also got my regular audio CDRs for home/car listening, and the original DAT as an "emergency backup".

It does seem a bit extreme/obssesive compulsive, but it's nice to know at least one of the many backups should be good in case of HDD/media failure.

[bgalizio]

I do the following for backups:

Master WAV with cue sheet > DVD
Master FLAC > DVD
Final FLAC > DVD, 2nd hard drive in PC, BT/archive
Audio CD > big booklet for home, copy for the car, copies for friends/family

I figure that has me covered. If all my optical media fails, hopefully my hard drive is fine. If that fails too, someone else ought to have the tracked FLAC from BT or archive. If everything fails, then I guess I'm screwed.

EDIT: I keep the master on the JB3 until I have all of the above covered.

[Brian Skalinder]

I've heard horror stories of things going badly wrong with friends' RAID arrays. Enough that my personal preference would be to use optical media / external HDD / offsite for backup instead.

True enough.  And that's why RAID should be only one layer of a more comprehensive backup strategy.  For me, my HD also serves as my primary playback source, so if RAID works, it'll save me the hassle of re-loading all my FLACs from optical media back onto the HD.

Equally, I've heard lots of success stories with RAID. If you decide to use RAID, it goes without saying you should test thoroughly! Are the drives hot-pluggable? Pull one out, pop in a new one and make sure you can actually recover the data!

Great point - this is sooooo important, and few people actually fully test their backup strategy, assuming it will work the way they expect.

You don't have to go through all that complexity for #4.  Just do two of #1 and/or #2 and keep a set of the discs off site. What's the big deal?

For me, anyway, [4] isn't a big deal (at least up front, anyway) - the off-site backup is simply a positive side effect of an action I'm already taking:  sharing the music.  On the back end, of course, it'd prove a big deal re-acquiring all those recordings.  I could back up the discs and move off-site myself, but I know I'd have a tough time keeping up with it - I know I'd do it too infrequently.  And that would actually act as a fifth backup strategy.  It's pretty unlikely [1-3] will all fail at once, so I'm willing to accept the possibility of a big hit on the recovery side (the hassle of re-acquiring from fellow music lovers), especially since I know I'd have a tough time keeping up with the off-site disc storage.  If I didn't widely share my recordings, you better believe I'd store my own personal backup discs off-site!

Good feedback, and I'm glad to see others are thinking about this issue, and ensuring they have appropriate backups.  As Nick's post suggests - it only takes a single problem scenario to motivate one into establishing a proper backup strategy.

[tapeworm48]

if people are burning original WAVs to DVDs, is it worth investing in something like www.audio-dvd-creator.com so that the DVDs can be played with your home system, and not just on the computer?  i figure that prices for portable dvd players for your car will drop (or maybe they already have) to the point where its affordable, and could replace cd players.

[Brian Skalinder]

if people are burning original WAVs to DVDs, is it worth investing in something like www.audio-dvd-creator.com so that the DVDs can be played with your home system, and not just on the computer?

I hadn't thought of that.  I'm not sure how DVD-audio works, but I know with WAV files stored as Redbook CD-audio you lose error-handling capability v. WAV files stored as data retaining some error-handling.  If it's the same with DVD, I'd be leary of burning in DVD-audio format for backup.  Not to mention DAE is required, then, to access the actual WAV files.

i figure that prices for portable dvd players for your car will drop (or maybe they already have) to the point where its affordable, and could replace cd players.

I suspect DVD-audio won't catch on in a big way for cars, but rather (or if DVD-audio catches on in cars, in addition) car stereos will include inputs for portable media players.  This has already happened, and I think the trend will continue.

[Brian]

for those of you who backup a recording untracked(in either .wav or .flac.....more especially .wav)

what's your reasoning behind this?

I ask because It seems to me that an optical and HDD of both the tracked .wav files and flac files is enough(along with any audio discs you burn to listen to outside of your computer).  Why is it important to have a backup of a file that will include useless audio(read: the audio you cut out for your mastered recording).

Is it just beacuse the raw wave file was the "original" file recorded and that it is somehow less likely to be prone to error than the edited, tracked .wav version?

thanks!

BTW - this is a great and VERY important thread.  It should be stickied at the top here or at least in the archive.  Just suggestion as I think this is the most over-looked aspect of recording.

[tapeworm48]

I hadn't thought of that.  I'm not sure how DVD-audio works, but I know with WAV files stored as Redbook CD-audio you lose error-handling capability v. WAV files stored as data retaining some error-handling.  If it's the same with DVD, I'd be leary of burning in DVD-audio format for backup.  Not to mention DAE is required, then, to access the actual WAV files.

i see what you're saying.  i didnt know there was a difference between WAV files burned as data vs. audio.  so extraction is not required then to move WAV files from a backup CD/DVD to your hard drive?

btw from what i have been told, audio-DVD is different from DVD-Audio.  The former is a DVD with no video present, and thus can be played on any DVD player, whereas DVD-A requires a player capable of playing DVD-A discs.

[Brian Skalinder]

for those of you who backup a recording untracked(in either .wav or .flac.....more especially .wav)

what's your reasoning behind this?

That's a really good question, and one I've never really given much thought.  For me, I think it's a carryover from DAT - gotta have my master recording, untouched.  But now that I think about it - archiving a big file...if any part of the optical media goes bad, I may lose the entire file.  On the other hand, if I do any substantial editing - I want that single, big master file available in case down the road I wish to edit differently.

Is it just beacuse the raw wave file was the "original" file recorded and that it is somehow less likely to be prone to error than the edited, tracked .wav version?

The original master definitely isn't less prone to error.  In fact, since it's a single, big file...it may be more difficult to recover in the event of partial damage.  At least on the tracked backups, damage may be limited to a single track.

BTW - this is a great and VERY important thread.  It should be stickied at the top here or at least in the archive.  Just suggestion as I think this is the most over-looked aspect of recording.

Agreed.  The intent here, driven by Wayne's comment on another thread, is to gather information on what people are doing, and then sticky or archive it (or both).

i see what you're saying.  i didnt know there was a difference between WAV files burned as data vs. audio.

Definitely a difference, at least burning as CD audio.  Not so sure about DVD, but I suspect so.

so extraction is not required then to move WAV files from a backup CD/DVD to your hard drive?

Correct - DAE is not required/performed when copying WAV data files from CD/DVD to HD.

btw from what i have been told, audio-DVD is different from DVD-Audio.  The former is a DVD with no video present, and thus can be played on any DVD player, whereas DVD-A requires a player capable of playing DVD-A discs.

Good clarificaiton - I'm not into DVD-audio or audio-DVD burning, since I'm in the process of moving straight from CD to PC-based playback.  I'd forgotten there are two, very different options.

[sleepypedro]

btw from what i have been told, audio-DVD is different from DVD-Audio.  The former is a DVD with no video present, and thus can be played on any DVD player, whereas DVD-A requires a player capable of playing DVD-A discs.

Good clarificaiton - I'm not into DVD-audio or audio-DVD burning, since I'm in the process of moving straight from CD to PC-based playback.  I'd forgotten there are two, very different options.

before anybody else gets excited about http://www.audio-dvd-creator.com/ like i did, please be aware of its limitations:

You can choose the audio format from high quality (up to 6 hours PCM 48kHz/16bits) or high quantity (up to 45 hours AC3 192kbps).

if it can't do 24/96, as hillary duff once wisely noted, "that's so yesterday"

[nickgregory]

before anybody else gets excited about http://www.audio-dvd-creator.com/ like i did, please be aware of its limitations:

You can choose the audio format from high quality (up to 6 hours PCM 48kHz/16bits) or high quantity (up to 45 hours AC3 192kbps).

if it can't do 24/96, as hillary duff once wisely noted, "that's so yesterday"

actually not the case, I burn 24/48 and 24/96 discs all the time with Audio DVD Creator, the limitation it has is not a program limitation but rather a format limitation in that DVD-V will not support any sample rates other than 48 and 96