Topic: Archiving master WAVs to FLAC?

[dgale]

you've mentioned a couple of times the advantage of FFP's and md5's to verify, bit for bit, the integrity of FLAC files.  and that's definitely a good thing and should be done.  But there's nothing that says md5 checksums are exclusive to FLAC or SHN.   personally, I create an md5 checksum for all my "master" .WAV files before I burn to data DVD.  After they are burned, I verify the md5 and I know that the burned copy is identical, bit for bit.

That's true and point well taken, although FLAC and SHN files are data files and if I recall correctly from trying to understand the purpose and benefits of SHN over WAV in the early days of SHN, data files more readily handle copying, moving around in Windows etc. then do audio WAV files - perhaps I am mistaken about this but that was my recollection from when their purpose was first explained to me...back in the day when you had audio CDs and wanted to copy them, you had to extract the WAVs via EAC or similar extraction program (at least if you wanted to minimize errors you did it this way) to get them from CDR to HD, while SHN (and subsequently FLAC) files could be dragged and dropped in Windows Explorer.  Of course you could do this with WAV files as well but my understanding was they were more prone to having errors introduced doing this over when you properly extracted them, while it was much more of a non-issue with data files.  Someone much more in the know about such computer format issues should chime in here.  Basically, it was explained to me that the benefits of SHN/FLAC were three-fold - lossless compression (which was more of a big deal IMO back in the days before DVD-Rs and huge HDs), ffp/md5 (and yes you can create md5s for any files), and the fact that they were converted to data files instead of WAV audio format.   

[rowjimmytour]

I guess I have a different perspective on this - once you burn the wav file to DVD, transfer it to another HD or otherwise move or copy it, how do you confirm that this move/process occurred in such a way that the new file (yes, it's actually a new file now that you've moved it, copied it, burned it etc. - it is no longer the "master" WAV file once you move or alter it from the drive or card on which it was originally recorded) it a bit-for-bit accurate copy of the original file?  In the somewhat older days people when to extra lengths to use EAC to ensure that as WAV files were burned to discs and then extracted later that this was done without any uncertainty as to the integrity of the file as it is burned/extracted along the way.  Why would burning a master WAV to DVD and then at some point in the future pulling it back off that DVD for some sort of use be any different? 

When you convert your master WAV>FLAC, you get ffps (and if you choose to create them, MD5s as well) that you can use to verify that the file remains bit-for-bit accurate as it is burned to disc, pulled off the disc, copied to drives, transfered electronically etc.  Seems like most people take the ffps for granted these days and don't even bother to test them as files are copied/transfered, which is a mistake as that is the point.  Personally I could care less that FLAC compresses the files to save space - I use it for the certainty that the files remain true to the original source. 

I used to be paranoid and would back-up my 24 WAV files both as a FLAC on DVD and also just burn the WAV straight to DVD, until I realized it was pointless IMO and that down the road I would never use the WAV version - I would always use the FLAC version as I could run test the ffps, run the MD5s and know that the file had not been corrupted in anyway.

My main reason to burn the untouched wav files to DVD-R is for matrix and MS recordings but I still do them all. I figure I might be able to come up w/ a better mix or edit later from my self or some one else so I can easily access these files and mix or edit away.
Peace

[JasonSobel]

you've mentioned a couple of times the advantage of FFP's and md5's to verify, bit for bit, the integrity of FLAC files.  and that's definitely a good thing and should be done.  But there's nothing that says md5 checksums are exclusive to FLAC or SHN.   personally, I create an md5 checksum for all my "master" .WAV files before I burn to data DVD.  After they are burned, I verify the md5 and I know that the burned copy is identical, bit for bit.

That's true and point well taken, although FLAC and SHN files are data files and if I recall correctly from trying to understand the purpose and benefits of SHN over WAV in the early days of SHN, data files more readily handle copying, moving around in Windows etc. then do audio WAV files - perhaps I am mistaken about this but that was my recollection from when their purpose was first explained to me...back in the day when you had audio CDs and wanted to copy them, you had to extract the WAVs via EAC or similar extraction program (at least if you wanted to minimize errors you did it this way) to get them from CDR to HD, while SHN (and subsequently FLAC) files could be dragged and dropped in Windows Explorer.  Of course you could do this with WAV files as well but my understanding was they were more prone to having errors introduced doing this over when you properly extracted them, while it was much more of a non-issue with data files.  Someone much more in the know about such computer format issues should chime in here.  Basically, it was explained to me that the benefits of SHN/FLAC were three-fold - lossless compression (which was more of a big deal IMO back in the days before DVD-Rs and huge HDs), ffp/md5 (and yes you can create md5s for any files), and the fact that they were converted to data files instead of WAV audio format.   

you are correct about the audio CD format, in that they have to be extracted with EAC or some other similar program, and that they are much more prone to errors.  But we're all talking about burning data CDs and data DVD's.  If you burn a .wav file to a data DVD (or CD), then it's just like burning any other file (FLAC or SHN or whatever), and you can copy the .wav files back to the HD with the "drag and drop", and it'll be just as error-free as any FLAC or SHN file.  The key here is to not burn an audio CD, but keep everything as a data disc.

[BlingFree]

> "i dont trust flac to be faithful to the concept of sound design"

I'm wondering if you mean Sound Design like the 722? IN which case I understand. A wav file transfered from a 722 and then FLAC'd right away produces errors while encoding. I'm assuming because SD puts non-musical data in the wav file for later. Time code maybe or other data. I'm not sure and personally would be happy if it didn't do that anymore.

So in essence the md5s, in this case, wouldn't match between the original and the FLAC'd because of the missing data bits.

[morst]

if an md5 checks out on a master wave and an uncompressed flac then... no loss, right? thus making it safe and maybe even practical to archive to FLAC??

If you want to be really sure, you could create an MD5 for each original WAV file, then losslessly compress using FLAC, but SAVE THE WAV's MD5 along with the FLAC files. That way whenever you decode the FLAC files, you can check the resulting WAV against the original MD5 signature to make sure it's still there.

Seems like overkill to me, but it's possible that it could help at some point. Of course if you want to be really finicky, you will want to decode each flac you encode, and check the MD5 before creating the archival backups, just to make sure it encoded correctly!

Personally, I do this:
1- copy original files to a hard drive on my computer
2- FLAC encode the files
3- load the original file into audacity and do any processing needed, including tracking. I prefer to store the audacity session on a separate hard drive from the original file, just in case.
4- burn 2 identical DVD-R discs with the FLAC's of the original unprocessed files (renamed for clarity) as well as the final output of the processing.
5- share the music. The more people have it, the less likely it is to disappear!

I need to start keeping my second set of backups in my storage unit, just in case! Really, making two backups is pretty much a RAID mirror.
By the way, I think DVD-R is more reliable for long term storage than DVD+R. The +R discs have faster burn rates, but the -R discs are supposed to last longer.

[DSatz]

As a reply to the last several messages in this thread: The advice to store the wave audio data as data files rather than in a directly playable audio disc format is well taken. The Orange Book standard for CD-ROM and all standards for DVD-ROM formats (DVD-R, DVD+R, etc.) include checksumming for the data of each sector. The Red Book standard for audio CDs does not. Basically if you can transfer a data file from a DVD-R, DVD+R or CD-ROM without encountering read error messages from your operating system, you should have the original data.

In addition, just focusing for a moment on CDs (audio versus CD-ROM), while the CD-ROM data format is based on the physical format of the audio CD (due to the fact that the medium was invented for audio use, and its use for data came only later when the audio medium had established itself), audio CDs were not designed to be "randomly accessed" nor necessarily to give bit-accurate reading. The logical data sectors on an audio CD aren't numbered; a player can only infer where it is reading from. This is why software such as EAC is necessary for high-speed digital audio extraction if you don't have a Plextor or other drive with firmware that has the equivalent logic built in. If the player's output data buffer fills up, the disc keeps spinning but the reading stops until there is room in the buffer. Then reading has to resume, but exactly where?

All disc media have this same problem, but the other ones (such as your hard drive) cope with it by numbering their logical sectors so that they can seek to a particular sector and "know" where they are. With audio CDs, the position can only be estimated and contents compared, etc., to establish exact head location. With CD-ROMs the logical sectors contain additional data which include sector numbers, so that precise seeking is possible and data are not repeated or skipped during an interrupted transfer (the normal kind at high speeds).

CD-ROM was designed as a distribution medium for software as well as a carrier for enterprise business data; undetected, uncorrected bit errors are of course unacceptable in those applications. In the playback of audio CDs a set of techniques known as "error concealment" based on psychoacoustic assumptions is used; these techniques cannot be used on CD-ROM. Thus an additional layer of error detection and error correction data is included in the Orange Book format, such that the bit error rate is reduced by a factor of 1,024 over audio CD, and the error detection is powerful enough that it is mathematically very unlikely to be fooled even once during the estimated service lifetime of a player.

--best regards

[morst]

music > microphones > a recorder of some sort

DSatz- your posts are always awesome. Glad you're here. +T bro!

[digifish_music]

Flac results in ~30-50(max)% compression. I am not convinced that it's worth the hassle given 500 Gb drives are so cheap...and as are DVDs?

digifish

[datbrad]

Also, when burning data DVDs, remember there is a data verification step where the freshly burned disc is immediately read and the data on it compared to the source file. If there are any differences, the disc ejects with a message that the disc "failed data verification". If you get the "successful" message with a green check mark or whatever, this means the file on the DVD matches the data of the original file exactly.