Topic: FLAC Encoding Experiment: Is FLAC really lossless? (long)

[splumer]

In several internet forums, I have seen some people mention that FLAC encoding actually is lossy, but in amounts that are barely measurable, let alone noticeable by a listener. I suspected that hypothesis might be true, so to test it, I decided to repeatedly encode and decode a constant tone, and then analyze both the amplitude and frequency after several cycles of encoding and decoding, representing several "generations" of encoding. I used a standard tone generator used to test sound equipment, run through a Mackie mixer into a USB audio interface and into a laptop PC running Cool Edit Pro.

The control was a 1000 Hz tone, which, under frequency analysis in Cool Edit, had an actual frequency of 977.09 Hz. The deviation from 1000 Hz I suspect was due to tone coloration of the mixer, despite the EQ controls being set to zero. In the end, though, it doesn’t matter, because the change in frequency or amplitude over successive encodings is what is being measured. The initial peak amplitude was set to be about -3 db, which the statistical analysis function of CEP measured at -2.82 db.

With those baselines established, I encoded the .wav file of the control tone to FLAC level 6 using Trader’s Little Helper. After every fifth encoding/decoding cycle, I opened the .wav file in CEP again, and did the frequency and statistical analysis. Even after 20 cycles, there was no change in amplitude. Frequency, however, had an interesting occurrence: between the fifth and tenth cycles, the frequency went up to 978.57 Hz, a change of 0.15 percent. This frequency remained the same through the fifteenth cycle, but went down to the original value of 977.09 Hz by the twentieth. Suspecting an error on my part, I started the experiment over.

This time, the data remained the same throughout all 20 cycles. I noticed, however, that the frequency reported by the frequency analysis changed, based on how much of the waveform was highlighted when I ran the frequency analysis. Armed with this knowledge, I returned to the data from the first experiment. This time, I highlighted the whole waveform every time I ran the frequency analysis, as I had done during the second experiment. This time, the frequency remained unchanged.

Conclusion: FLAC encoding does not change the amplitude or frequency of a waveform within the number of encodings that would be normally encountered. Therefore, I feel safe in stating that FLAC encoding is indeed wholly lossless.

[Javier Cinakowski]

Seems like you put a lot of thought into this.  Can I ask why you didn't consider using checksum files at all in your experiment.  My point is that there is a lot more to digital audio than frequency and amplitude, why not evaluate the actual data (the 1's and 0's).  Take a .wav file and make a checksum (.md5), then encode to flac, encode back to .wav, and verify that it matches the original.  This will verify that every bit of information is retained in the flac file, down to the very last binary digit.   

[Javier Cinakowski]

The control was a 1000 Hz tone, which, under frequency analysis in Cool Edit, had an actual frequency of 977.09 Hz. The deviation from 1000 Hz I suspect was due to tone coloration of the mixer, despite the EQ controls being set to zero.

My guess is that the variation you observed was due to the input impedience of your mixer.  Those tone generators are calibrated to a specific input impedience.  Lots of preamps vary in this spec...

[burris]

The people who are saying FLAC is lossy are full of it.  Ask them to cough up a piece of data that will not encode and then decode to the same data.  FLAC includes the tools to test this.  When you encode, it calculates the MD5 hash of the incoming data and stores that in a metadata block as the fingerprint.  When you run flac in test mode (flac -t) it decodes the file and calculates the hash of the outgoing data, then compares that with the fingerprint of the original.  If its the same then there is no loss and flac says "OK."  Even if you don't trust the MD5 hash function, its easy to do a bit-by-bit comparison.

You can encode anything with FLAC, including .exes or video, and you'll always get the same data back out.  The catch is you won't get good compression on anything but audio.  It is a lossless algorithm similar to the ubiquitous ZIP, but optimized for audio.  The failure mode of a codec like Zip or FLAC is to have a poor compression ratio, not output bogus data.  Its possible there is a bug, however, which is one reason why flac has a test mode.  If you do find a bug, report it and it will get fixed.  However, to say that FLAC is lossy in any way and that the loss is imperceptible is simply incorrect.

[twatts (pants are so over-rated...)]

The people who are saying FLAC is lossy are full of it. 

yeah, right...  If you had these cables, you'd be able to hear the difference...   

http://www.noiseaddicts.com/2008/11/most-expensive-speaker-cable-world-audioquest-audiophile/

Terry

[splumer]

Seems like you put a lot of thought into this.  Can I ask why you didn't consider using checksum files at all in your experiment.  My point is that there is a lot more to digital audio than frequency and amplitude, why not evaluate the actual data (the 1's and 0's).  Take a .wav file and make a checksum (.md5), then encode to flac, encode back to .wav, and verify that it matches the original.  This will verify that every bit of information is retained in the flac file, down to the very last binary digit.

It's been done, that's why. Plus, I have to admit I don't fully understand and thus don't fully trust, checksums. It's also possible that there can be changes in, say, file header info that would result in a different checksum without any change to the audio. My experiment was to test "audio quality," in quote because it's such a subjective issue.

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

[kirk97132]

Why not use level 8 encoding the highest quality? 

[Javier Cinakowski]

Why not use level 8 encoding the highest quality?

Increasing the flac level doesn nothing to change the actual audio quaility.  It just takes longer to encode with the benefit of smaller files.  A decoded level 4 flac and a decoded level 8 flac are identical...

[Gutbucket]

I suspect the small frequency analysis variations are simply below tolerance artifacts of CEP's frequency and statistical analysis algorithm.

[scb]

It's also possible that there can be changes in, say, file header info that would result in a different checksum without any change to the audio.

For a full file MD5, yes, but not a flac fingerprint.  A flac fingerprint, like the shntool "st5", is a hash of just the PCM audio data in a file.  You can change a file's header all you want (hell, you can even remove the header) and the fingerprint or shntool st5/md5 will be exactly the same if the audio didn't change. 

[anonymous_user]

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

There would be no reason at all to do that. Burning/ripping CD's is absolutely not lossless in the manner in which you are speaking, and has really never been advertised as such. Everytime you convert/burn from wav to cda, it is not a truly lossless conversion such as going from wav to flac or vice versa, not at all. So yes, if you were to do it over the course of multiple generations, there would be noticable differences, and I don't think you would even need to use frequency/amplitude analysis tools to notice them, especially if it was 20 generations apart. I'm pretty sure most people are aware of that. It's one reason why using flac, archiving, and EAC is so widespread.

It obviously doesn't apply to burning/copying DVD's though.

[Brian Skalinder]

I have to admit I don't fully understand and thus don't fully trust, checksums.

If you don't trust checksums, you could use a tool to compare bit-for-bit the audio before and after FLACing.  Foobar2000 + Binary Comparator plugin will do the trick.  Though if you don't trust checksums, you probably won't trust the Binary Comparator, either.

[live2496]

If doing comparisons you would be better off doing byte to byte comparisons of original and decoded files. But if the MD5 checksum is the same you can assume that the files are identical.

[splumer]

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

There would be no reason at all to do that. Burning/ripping CD's is absolutely not lossless in the manner in which you are speaking, and has really never been advertised as such. Everytime you convert/burn from wav to cda, it is not a truly lossless conversion such as going from wav to flac or vice versa, not at all. So yes, if you were to do it over the course of multiple generations, there would be noticable differences, and I don't think you would even need to use frequency/amplitude analysis tools to notice them, especially if it was 20 generations apart. I'm pretty sure most people are aware of that. It's one reason why using flac, archiving, and EAC is so widespread.

It obviously doesn't apply to burning/copying DVD's though.

But EAC stands for Exact Audio Copy, and thus should be exact. Since it's a digital conversion process, there really is no reason why it couldn't be lossless. I've never seen a demonstration of it one way or the other, though.

The checksum comparison has been done, though. Maybe I was the peer review on that one? 

I suspect the small frequency analysis variations are simply below tolerance artifacts of CEP's frequency and statistical analysis algorithm.

But there weren't any variations. When the whole waveform was highlighted and the analysis run, it always gave the same frequency. An average, perhaps? Variations based on individual points along the waveform could be due to a variety of factors, from voltage fluctuations, temperature changes, etc., especially when you're talking about variations of thousandths of Hertz.

Thanks for all the comments!

[H₂O]

When talking CD's you are talking about bit perfect not lossless.  CD's use a basic CRC for error checking which may pass with a non- bit perfect transfer.  With md5 and even sha1 hashing collisions can occur (a situation where A hash matches even though the data does not - although the larger the hash the less likely this is to happen).  The only way to know for sure is a bit by bit comparison.

Most cd burners skip through errors (even Most dat decks skip through errors). The problem I see with CD transfers is that most discs is that most people doing the copies back in the day didn't care about quailty only quantity so alot have no lineage and where done in a very hasty fashion. 

Although CD copies are not lossy compressed many are very far from bit perfect!


Lossy/Lossless refers to compression


bit perfect refers to how accurate a copy is


lossy compression is inherently not bit perfect

[Brian Skalinder]

The checksum comparison has been done, though. Maybe I was the peer review on that one? 

I have to admit I don't fully understand and thus don't fully trust, checksums.

< scratches head >

[H₂O]

An exert from about CD-DA and error handling from http://makbit.com/articles/cd-overview.pdf


Error Handling:
· The goal of error handling on a CD-DA is the detection and correction of typical
error patterns. An error is usually the result of scratches or dirt.
· The first level of error handling implements two-stage error correction according
to the Reed-Solomon algorithm. For every 24 audio bytes, there are two groups
of correction data bytes, each of four bytes.
· The first group corrects single-byte errors while the second group corrects doublebyte errors.
· In the second level, real consecutive data bytes (24 bytes) are distributed over
multiple frames. The audio data are stored interleaved on the CD-DA. In this
way, burst errors will always damage only parts of the data.
· An error rate of 10-8
 is achieved.


So it's a bit more complex than a standard CRC but still as mentioned above there is still the possibility of having undetected or correctable errors at 10^-8

[anonymous_user]

But EAC stands for Exact Audio Copy, and thus should be exact. Since it's a digital conversion process, there really is no reason why it couldn't be lossless. I've never seen a demonstration of it one way or the other, though.

Let me clarify. It's not so much going from CD to wav, it's the opposite. So yes, in theory, EAC, or any CD ripping process, is supposed to give you both a bit-perfect and lossless copy of what is on the disc, and I personally tend to agree this is the case. The difference is that EAC (as opposed to any other CD ripping process) is really the only thing that can ensure a CD is 100% extracted correctly and properly, and 100% error free and completely identical to the original.

The above posts about CRC and error-checking etc I think aren't really relevant directly, because for one that wasn't what I was referring to, things like errors can be avoided with EAC anyway. So for two, I was assuming the best possible scenarios, which would be using EAC for completely perfect extraction, no errors on either the discs or on the extraction, and matching CRC's, etc and anything else I'm forgetting.

It's the opposite process, going from wav to CD, that isn't quite lossless in the way you were originally asking about. And as stated in one of the posts above, there is a difference between what is "lossless" and technically lossless. Whatever phrasing you want to use, most people use the term lossless to refer to no compression, and no frequencies lost, things like that, ie: as opposed to mp3's and compression. But saying something is 100% lossless is different, that would mean that there's absolutely no change whatsoever. This is why I used certain phrases in my original post. I said things like "is not lossless in the manner in which you are referring to", "not truly lossless", and "not like going from wav to flac or vice versa" to differentiate it from just saying "lossless". It might be lossless in terms of no compression or frequencies being lost, but that's probably where it ends. I'm not going to pretend to be an expert on how it works or why it happens, I just know that if you copy a CD enough generations apart, even if they're all competely perfect extractions, there would be a difference after a given number of enough copies.

Also, to answer your quoted text directly, keep in mind it's not quite the simple or straight forward digital conversion process that you may think. Remember that when you extract a CD to wav, the wav files are much larger filesize than the CD itself, and again, when you burn wav files to an audio CD, the filesize of the CD ends up being much smaller than the wav's themselves. An 80-minute CDr only holds about 700mb or 703mb of data. That same 80-minute CDr extracted to wav files is much larger. That's why the files aren't compatible to begin with, they're not the same, and why extracting an audio CD generally takes longer than just copying files from a data CD.

[twatts (pants are so over-rated...)]

When talking CD's you are talking about bit perfect not lossless.  CD's use a basic CRC for error checking which may pass with a non- bit perfect transfer.  With md5 and even sha1 hashing collisions can occur (a situation where A hash matches even though the data does not - although the larger the hash the less likely this is to happen).  The only way to know for sure is a bit by bit comparison.

Most cd burners skip through errors (even Most dat decks skip through errors). The problem I see with CD transfers is that most discs is that most people doing the copies back in the day didn't care about quailty only quantity so alot have no lineage and where done in a very hasty fashion. 

Although CD copies are not lossy compressed many are very far from bit perfect!


Lossy/Lossless refers to compression


bit perfect refers to how accurate a copy is


lossy compression is inherently not bit perfect

CDBurning isn't always perfect...  EACing a burned CDR may give you a perfect copy of the CDR, but if the data was faulty, so is your new rip. 

Those of us that are old school ETREE saw it happening when people would EAC an already circulating SHN>CDR and the MD5s wouldn't match up.  Lossless?  doubtful...  Bit Perfect?  Most likely...  But there was a change in the data that is shown with the MD5. 

Considering what it is, FLAC is the "best" at managing data so there is little to no audio information change/loss...

Terry

[H₂O]

Actually EAC even in secure mode may not rip a bit accurate copy even if it says there is no errors - that's why the have added online Checksum lookups to EAC. 


I have had a number of tracks on Retail CD's that give no errors in Secure extraction but fail the lookup -> these tracks will also give different Checksums in different drives.


For live CD's since they would not be stored on the online DB there is no real way to guarantee an EAC rip is bit accurate.


At that level though flaws are typically inaudible.

[live2496]

If EAC reports the copy as 100%, then it is bit perfect.

To get exact copies with EAC you have to have determined (and set) both the read offset and the write offset of the drive. This is covered in the FAQ on the web pages. I have done this and verified byte for byte copies.

If read offsets and write offsets are not set you can get discrepencies in the first sector which will be padded with binary zeros. This can cause different MD5 checksums even though the extracted audio otherwise matches.

Once read offsets and write offsets are set, and you burn using EAC and the same cue sheet you will get a bit exact copy. But your media and burner must be good so that EAC can extract the data off the disc with 100% accuracy.

If there is a bad scratch on the disc or some imperfection then EAC may not be able to guarantee that the copy is 100% accurate.  But the error report will indicate that.

[twatts (pants are so over-rated...)]

If EAC reports the copy as 100%, then it is bit perfect.

To get exact copies with EAC you have to have determined (and set) both the read offset and the write offset of the drive. This is covered in the FAQ on the web pages. I have done this and verified byte for byte copies.

If read offsets and write offsets are not set you can get discrepencies in the first sector which will be padded with binary zeros. This can cause different MD5 checksums even though the extracted audio otherwise matches.

Once read offsets and write offsets are set, and you burn using EAC and the same cue sheet you will get a bit exact copy. But your media and burner must be good so that EAC can extract the data off the disc with 100% accuracy.

If there is a bad scratch on the disc or some imperfection then EAC may not be able to guarantee that the copy is 100% accurate.  But the error report will indicate that.

What he said...  For myself, I could never get my offsets set properly and quit trying...  I hardly EAC anything now, and if I do, its not important enough to be THAT accurate...

Terry

[H₂O]

If EAC reports the copy as 100%, then it is bit perfect.

To get exact copies with EAC you have to have determined (and set) both the read offset and the write offset of the drive. This is covered in the FAQ on the web pages. I have done this and verified byte for byte copies.

If read offsets and write offsets are not set you can get discrepencies in the first sector which will be padded with binary zeros. This can cause different MD5 checksums even though the extracted audio otherwise matches.

Once read offsets and write offsets are set, and you burn using EAC and the same cue sheet you will get a bit exact copy. But your media and burner must be good so that EAC can extract the data off the disc with 100% accuracy.

If there is a bad scratch on the disc or some imperfection then EAC may not be able to guarantee that the copy is 100% accurate.  But the error report will indicate that.

Not True!  EAC is not guaranteed to be bit perfect but it is as close as you can get - as it does Error Checking and correction and can compare calculated checksums with an online database to be more accurate than CD-DA supports.


I have aligned the offset in the settings and have recieved no errors but the checksum look up fails it will say "Track read without errors but checksum does not match online checksum"  On these tracks I get different calc'd checksums on different drives (there are only very light buffs on the disc).


The likelyhood increases if the track quality read reports closer to 100%

[live2496]

Admittedly, I have only checked original discs burned with my drive, not against any CDDB entries.

But I have been able to get 100% accurate burns and byte compare against the original 16-bit data.

I must be wrong about 100% meaning bit accurate from what you said. But my experience has been very good with my own discs and burns/extracts made with EAC. I'm using Taiyo Yuden media too and that probably helps. Plus the extracts are done right after burning the disc.

[H₂O]

I have ripped over 1000 retail CD's using EAC and probably 200-300 since they added the online checksum lookup.  I have only seen EAC report no errors but an incorrect checksum on 2-3 tracks of those 200-300 CD's so it is extremely rare. 


10 tracks per CD 300 CD's = 3000 tracks 3 false positive or about 0.1% so it's very rare.

[Will_S]

I just know that if you copy a CD enough generations apart, even if they're all competely perfect extractions, there would be a difference after a given number of enough copies.

Err, no (as long as the burns are also error free).  It's just that with enough generations, one is likely not to be a "completely perfect extraction".

Also, to answer your quoted text directly, keep in mind it's not quite the simple or straight forward digital conversion process that you may think. Remember that when you extract a CD to wav, the wav files are much larger filesize than the CD itself, and again, when you burn wav files to an audio CD, the filesize of the CD ends up being much smaller than the wav's themselves. An 80-minute CDr only holds about 700mb or 703mb of data. That same 80-minute CDr extracted to wav files is much larger. That's why the files aren't compatible to begin with, they're not the same, and why extracting an audio CD generally takes longer than just copying files from a data CD.

This is not quite right either.  Properly extracted wav files and the audio data on a CD contain the exact same data and are in fact the same size.  What can get confusing is when you try to match up the size of say an 80 minute wav file with the data capacity of a CD-R that can hold 80 mins of music.  That same CD, when burned as a data CD can only hold ~700 MB of data, which is smaller than an 80 minute wav file.  However that is because of the different ways that data are written to audio CDs vs data CDs.  All CDs are made up of sectors of 2,352 bytes.  In an audio CD, all 2,352 bytes of each sector contain audio data.  But in data CDs, each sector contains only 2,048 bytes of "data" in terms of file contents, with the remaining bytes consumed by header information and error-correcting codes.  See http://en.wikipedia.org/wiki/CD-ROM#CD_sector_contents.  That is why the wav files from a CD can take up more bytes than you can burn to a CD-ROM data CD, not because they contain a different amount of information. 

(An added complication is that you can overburn audio CDs, so that you can actually write more sectors to the same sized disc.  But the main reason you can get wav files totalling more than 700MB from an 80 min CD is the "loss" of part of the storage capacity of data CDs to header information and error correction that is not present in audio CDs.)

[faninor]

So FLAC is okay, but is ZIP or RAR lossless???

[twatts (pants are so over-rated...)]

So FLAC is okay, but is ZIP or RAR lossless???

Yes, but pointless.  ZIPing or RARing will not reduce the size of a WAV file as much as FLAC will...  But if you want to ZIP the FLAC File Folder for ease of FTP or something, there is no harm in that...  Tangentally, Archive.org delivers its file folders in ZIP format.  You unZIP and you have your FLAC Files...

Terry

[faninor]

You missed my sarcasm. 

I was just alluding to the fact that people use ZIP and RAR all the time make their text files, photos, and executables smaller for transferring from point to point with no loss of data without worrying about it. I've never seen anybody question that, but if we're talking about audio then the same information stored in two different ways becomes suspicious.

[SmokinJoe]

Let's go back to the statement "I don't understand and trust checksums".  That seems unusual to some of us, but I think it's a good educational opportunity.

Let me ask you this...  "if 2 wav files contain the same computer data, do they contain the same sound data?".  The answer is yes.  I'm not talking about compression or checksums or converters, just pointing out that comparing computer files is a simpler and more definitive test than analyzing data.  Either the files match or they don't match.  You could write a program to open both files, go down through them byte by byte and see if they match pretty easily.

So if I have a wave file, I can compress it with flac, then uncompress it to get another wave file.  Then I can compare the first wav file to the second wav file byte for byte and if they match, we didn't lose any data, hence the definition of lossless.   That becomes definitive proof.  I encourage you to try it yourself.  Do it over and over again until you believe it.

The problem with comparing files byte by byte is that it's not very efficient when it comes to transferring large files across the internet.   You would need 2 full size copies of data to compare, which means we have to transfer twice as much data.  So the math and computer geeks invented checksums.  It works something like this... let's say I call you on the phone and give you a bunch of numbers and you write them down...   34511234..... 134142412...... 2634563..... 37634673.... go on for 1000 numbers like that.  We could say "read them back to me", and we could double check number for number.  Another way to check them would be to say "add them up and you should get 12341412341".   If we both get the same result, we probably copied all the data over accurately, and it's efficient because we only had to send 1 extra number instead of all of them.  That algorithm isn't very robust, and you could make 2 mistakes that mask each other.  So the geeks think deeper and come up with some other math functions, where the odds of not detecting mistakes are "a bazillion to one".  The entire internet and everything computer related is based on "a bazillion to one is an acceptable risk".  More on checksums here.  http://en.wikipedia.org/wiki/Checksum

So, if you take a wave file, flac it, unflac it, and compare the checksums they will match.  The only way they won't match is if your computer is failing.  The odds your computer is failing is a million to one.  The odds of not detecting that failure with md5sum are a bazillion to one.  If you use Flac Fingerprints (ffp) and md5sums (md5) and flac checksums (st5) all together, the odds are a million times a bazillion cubed.  So checksums work, and they are universally accepted as such.

~~~~~~~~

So where did this whole thing about "lossy flac" scare come from?  Well, there was once a guy to proposed a "lossy flac" standard.  Bad idea. It never took off.
http://www.hydrogenaudio.org/forums/index.php?showtopic=55522

Then there was this guy who says "in foobar it says I'm playing at 455kb/s, but it should be playing at 1411kb/s".  From there he draws the wild conclusion that Flac is lossy, when the real answer is that foobar is only telling you how much data it has to uncompress on the fly.  http://www.anythingbutipod.com/forum/showthread.php?t=38885

The one thing I will concede as a possibility is that streaming flac programs might not be perfect.  If I put flac files on my rockboxed H120 and play out the optical to my stereo is that bit perfect compared to wav?  I dunno.  But that's another issue... that doesn't mean Flac is lossy, just the streaming software.