Topic: Fluckin' Flac!!!!

[MattD]

A drive error has nothing to do with the algorithm. Regardless of how what format your data is stored in, if you have a bad sector you generally have lost the data in it. As mentioned earlier, FLAC can decode through errors, so if the remainder of the file is still intact, then it's mostly recoverable. With a compressed file, one sector of compressed data holds more information than a sector of uncompressed data, so you'll lose more of the original wave if it's compressed.

With the move away from physical tapes, I urge people to realize that digital data essentially doesn't exist unless it exists in three distinct places. I generally have two synchronized hard drives, a binder of DVDs and I regularly FTP shows to a friend who also keeps them around. The latter is a good backup method: find a trusted friend who likes listening to the stuff you tape and tell them you'll give them a copy of everything you record under the condition that they keep it.

Using multiple backups has saved my ass in the past. Before I had a pair of drives, I had one die. "No problem," I thought, "I'll just use the backup CDs." Unfortunately for me, one of the CDs wasn't good; I could only pull about half of the files off it. I was able to track down a copy from someone I'd traded with nearly a year before. I learned my lesson then and instituted my current system. To save time from FTP transfers, my buddy recently mailed me a 120 GB drive. I filled it and will send it back this week.

[F.O.Bean]

With the move away from physical tapes, I urge people to realize that digital data essentially doesn't exist unless it exists in three distinct places. I generally have two synchronized hard drives, a binder of DVDs and I regularly FTP shows to a friend who also keeps them around. The latter is a good backup method: find a trusted friend who likes listening to the stuff you tape and tell them you'll give them a copy of everything you record under the condition that they keep it.

i also burnm the master flac sets untracked onto cd-r

but this is def the ebst way to make sure there are copies around

and also, usually, if a flac cd/dvd doesnt wanna copy verify, in my cases, it ahs only been 1-2 tracks and theyre usually easily found and easy to get 1-2 tracks rather than losing a whole show

[it-goes-to-eleven]

I've posted this before but it should be repeated... I monitor the health of all my drives using the built in SMART capability.

When I get a new drive, I immediately run a 'short' test on it (2 minutes) and then a 'long' test (about an hour). The drive tests itself - they are pretty smart these days. A drive I recently bought couldn't even pass the short test.

http://smartmontools.sourceforge.net

[pfife]

A drive error has nothing to do with the algorithm. Regardless of how what format your data is stored in, if you have a bad sector you generally have lost the data in it. As mentioned earlier, FLAC can decode through errors, so if the remainder of the file is still intact, then it's mostly recoverable. With a compressed file, one sector of compressed data holds more information than a sector of uncompressed data, so you'll lose more of the original wave if it's compressed.

Well, I have no problem defering to your expertise... but from what I've been told, if a FLAC header doesn't accurately describe the FLAC data, then you are hosed.  I don't think the same can be said about WAV.  Also, with my FLACs being hosed, I've lost everything.  You can't lose much more of a file than "all".

[it-goes-to-eleven]

FWIW, I have one of his bad flacs and it has a bad header.  I really can't say what condition the rest of the file is in.. But when I try to extract with -F, flac pukes right on the header and says it isn't a flac, game over. I could dig into the flac header format and try and reconstruct the header (maybe copy a header from a diff file, etc) but it would be a bitch.  WAV headers are very simple, I could reconstruct one in my sleep.  Though we should try the -F on some of the other problem flacs.

[MattD]

I hadn't considered an error in the header. Your best bet would be to reconstruct the header by looking at a known good header in a hex editor and extrapolating from that using the FLAC specification. So long as the data is there, I wouldn't call it lost. You might want to shoot an email to one of the FLAC developers asking how to go about recovery. I've gotten replies from Josh Coalson in the past.

[mmmatt]

Good news!!!!  I just replaced my ram and it seems as though all is working fine!  Please keep the info going on the flac fix though... I've got some multitrac masters that are crap unless I can figure out the issue.

Thanks to all that helped and also to bean.  ha!  T's to the thread (aka "a round for the house on me")

Matt

[sml42]

Oh wow, errors in the header = not good

Here is my backup strategy: I take the master .wav and compress to .flac, I master the show = a bunch more flac files. I wait til I have a bunch of shows - enough to fill a dvd. After I burn a dvd, I do a file-by-file comparison (comparing every byte in every file) between the newly-burned dvd and what's on my hard drive. Until it's archived on optical media, I keep a copy on my jb3, a copy on my laptop and a copy on my desktop machine. I've got a ton of dvds in a storage wallet, which I'm looking at now with some concern!

I'm curious.. what backup scheme do the professionals use, e.g. archive.org... Diana?

Edit: great news with the new ram... backatcha!

best regards,
stephen

[mmmatt]

Oh wow, errors in the header = not good

Here is my backup strategy: I take the master .wav and compress to .flac, I master the show = a bunch more flac files. I wait til I have a bunch of shows - enough to fill a dvd. After I burn a dvd, I do a file-by-file comparison (comparing every byte in every file) between the newly-burned dvd and what's on my hard drive. Until it's archived on optical media, I keep a copy on my jb3, a copy on my laptop and a copy on my desktop machine. I've got a ton of dvds in a storage wallet, which I'm looking at now with some concern!

I'm curious.. what backup scheme do the professionals use, e.g. archive.org... Diana?

Edit: great news with the new ram... backatcha!

best regards,
stephen

I recall her saying that they have a few levels of backup on HD.  The archive started as a test for long term hd storage... hopefully she'll chime in and say that all better.

I flac, and md5 all my masters, as well as my finished files and then copy to dvd and put into a binder... that is how I originally figured this little issue out.  My md5 didn't clear on one of my dvd backups, and I was like... oops.  Then I went back to the original data on the hd and the md5 cleared.  I then tested the flac files and some were bunk.  Then I was OK for awhile and then the shit hit the fan.

Matt

[wbrisette]

I'm curious.. what backup scheme do the professionals use, e.g. archive.org... Diana?

One of the things done is put things on DVD-RAM (basically you're using it as a DVD disc). DVD-RAM is the only format that has built-in error checking into the recording scheme (it's also one of the reasons why Zaxcom hasn't moved away from DVD-RAM as the only non-hard drive media the Deva writes to). We probably need to come up with some good practice backup techniques for all of this since a lot more people are moving to non-tape solutions for recording. Then we need to put those in the FAQ section.

Wayne

[Karl]

interesting...

[Diana Hamilton]

I'm curious.. what backup scheme do the professionals use, e.g. archive.org... Diana?

Edit: great news with the new ram... backatcha!

best regards,
stephen

I recall her saying that they have a few levels of backup on HD.  The archive started as a test for long term hd storage... hopefully she'll chime in and say that all better.

I'm not a techie so I can't give any specifics, sorry. I gather IA has at least 1 Bay Area copy of each thing plus a European copy (Amsterdam?). The moving images and the audio collections are currently in 2 different types of filing/storage methods. Later the audio will be the same as movies.

When there has been a catastrophic loss of a chunk of LMA stuff (as happened at the end of June), I think it's because items were moving off the upload disk onto a failing public d/l disk, and being lost before they had a chance to be mirrored.

[mmmatt]

thanks diana!