Topic: FLAC Encoding Experiment: Is FLAC really lossless? (long)

[splumer]

In several internet forums, I have seen some people mention that FLAC encoding actually is lossy, but in amounts that are barely measurable, let alone noticeable by a listener. I suspected that hypothesis might be true, so to test it, I decided to repeatedly encode and decode a constant tone, and then analyze both the amplitude and frequency after several cycles of encoding and decoding, representing several "generations" of encoding. I used a standard tone generator used to test sound equipment, run through a Mackie mixer into a USB audio interface and into a laptop PC running Cool Edit Pro.

The control was a 1000 Hz tone, which, under frequency analysis in Cool Edit, had an actual frequency of 977.09 Hz. The deviation from 1000 Hz I suspect was due to tone coloration of the mixer, despite the EQ controls being set to zero. In the end, though, it doesn’t matter, because the change in frequency or amplitude over successive encodings is what is being measured. The initial peak amplitude was set to be about -3 db, which the statistical analysis function of CEP measured at -2.82 db.

With those baselines established, I encoded the .wav file of the control tone to FLAC level 6 using Trader’s Little Helper. After every fifth encoding/decoding cycle, I opened the .wav file in CEP again, and did the frequency and statistical analysis. Even after 20 cycles, there was no change in amplitude. Frequency, however, had an interesting occurrence: between the fifth and tenth cycles, the frequency went up to 978.57 Hz, a change of 0.15 percent. This frequency remained the same through the fifteenth cycle, but went down to the original value of 977.09 Hz by the twentieth. Suspecting an error on my part, I started the experiment over.

This time, the data remained the same throughout all 20 cycles. I noticed, however, that the frequency reported by the frequency analysis changed, based on how much of the waveform was highlighted when I ran the frequency analysis. Armed with this knowledge, I returned to the data from the first experiment. This time, I highlighted the whole waveform every time I ran the frequency analysis, as I had done during the second experiment. This time, the frequency remained unchanged.

Conclusion: FLAC encoding does not change the amplitude or frequency of a waveform within the number of encodings that would be normally encountered. Therefore, I feel safe in stating that FLAC encoding is indeed wholly lossless.

[Javier Cinakowski]

Seems like you put a lot of thought into this.  Can I ask why you didn't consider using checksum files at all in your experiment.  My point is that there is a lot more to digital audio than frequency and amplitude, why not evaluate the actual data (the 1's and 0's).  Take a .wav file and make a checksum (.md5), then encode to flac, encode back to .wav, and verify that it matches the original.  This will verify that every bit of information is retained in the flac file, down to the very last binary digit.   

[Javier Cinakowski]

The control was a 1000 Hz tone, which, under frequency analysis in Cool Edit, had an actual frequency of 977.09 Hz. The deviation from 1000 Hz I suspect was due to tone coloration of the mixer, despite the EQ controls being set to zero.

My guess is that the variation you observed was due to the input impedience of your mixer.  Those tone generators are calibrated to a specific input impedience.  Lots of preamps vary in this spec...

[burris]

The people who are saying FLAC is lossy are full of it.  Ask them to cough up a piece of data that will not encode and then decode to the same data.  FLAC includes the tools to test this.  When you encode, it calculates the MD5 hash of the incoming data and stores that in a metadata block as the fingerprint.  When you run flac in test mode (flac -t) it decodes the file and calculates the hash of the outgoing data, then compares that with the fingerprint of the original.  If its the same then there is no loss and flac says "OK."  Even if you don't trust the MD5 hash function, its easy to do a bit-by-bit comparison.

You can encode anything with FLAC, including .exes or video, and you'll always get the same data back out.  The catch is you won't get good compression on anything but audio.  It is a lossless algorithm similar to the ubiquitous ZIP, but optimized for audio.  The failure mode of a codec like Zip or FLAC is to have a poor compression ratio, not output bogus data.  Its possible there is a bug, however, which is one reason why flac has a test mode.  If you do find a bug, report it and it will get fixed.  However, to say that FLAC is lossy in any way and that the loss is imperceptible is simply incorrect.

[twatts (pants are so over-rated...)]

The people who are saying FLAC is lossy are full of it. 

yeah, right...  If you had these cables, you'd be able to hear the difference...   

http://www.noiseaddicts.com/2008/11/most-expensive-speaker-cable-world-audioquest-audiophile/

Terry

[splumer]

Seems like you put a lot of thought into this.  Can I ask why you didn't consider using checksum files at all in your experiment.  My point is that there is a lot more to digital audio than frequency and amplitude, why not evaluate the actual data (the 1's and 0's).  Take a .wav file and make a checksum (.md5), then encode to flac, encode back to .wav, and verify that it matches the original.  This will verify that every bit of information is retained in the flac file, down to the very last binary digit.

It's been done, that's why. Plus, I have to admit I don't fully understand and thus don't fully trust, checksums. It's also possible that there can be changes in, say, file header info that would result in a different checksum without any change to the audio. My experiment was to test "audio quality," in quote because it's such a subjective issue.

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

[kirk97132]

Why not use level 8 encoding the highest quality? 

[Javier Cinakowski]

Why not use level 8 encoding the highest quality?

Increasing the flac level doesn nothing to change the actual audio quaility.  It just takes longer to encode with the benefit of smaller files.  A decoded level 4 flac and a decoded level 8 flac are identical...

[Gutbucket]

I suspect the small frequency analysis variations are simply below tolerance artifacts of CEP's frequency and statistical analysis algorithm.

[scb]

It's also possible that there can be changes in, say, file header info that would result in a different checksum without any change to the audio.

For a full file MD5, yes, but not a flac fingerprint.  A flac fingerprint, like the shntool "st5", is a hash of just the PCM audio data in a file.  You can change a file's header all you want (hell, you can even remove the header) and the fingerprint or shntool st5/md5 will be exactly the same if the audio didn't change. 

[anonymous_user]

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

There would be no reason at all to do that. Burning/ripping CD's is absolutely not lossless in the manner in which you are speaking, and has really never been advertised as such. Everytime you convert/burn from wav to cda, it is not a truly lossless conversion such as going from wav to flac or vice versa, not at all. So yes, if you were to do it over the course of multiple generations, there would be noticable differences, and I don't think you would even need to use frequency/amplitude analysis tools to notice them, especially if it was 20 generations apart. I'm pretty sure most people are aware of that. It's one reason why using flac, archiving, and EAC is so widespread.

It obviously doesn't apply to burning/copying DVD's though.

[Brian Skalinder]

I have to admit I don't fully understand and thus don't fully trust, checksums.

If you don't trust checksums, you could use a tool to compare bit-for-bit the audio before and after FLACing.  Foobar2000 + Binary Comparator plugin will do the trick.  Though if you don't trust checksums, you probably won't trust the Binary Comparator, either.

[live2496]

If doing comparisons you would be better off doing byte to byte comparisons of original and decoded files. But if the MD5 checksum is the same you can assume that the files are identical.

[splumer]

Next, I may try multiple generations of burning/ripping CD's. Not sure if I want to use up a bunch of CD-R's for it, though.

There would be no reason at all to do that. Burning/ripping CD's is absolutely not lossless in the manner in which you are speaking, and has really never been advertised as such. Everytime you convert/burn from wav to cda, it is not a truly lossless conversion such as going from wav to flac or vice versa, not at all. So yes, if you were to do it over the course of multiple generations, there would be noticable differences, and I don't think you would even need to use frequency/amplitude analysis tools to notice them, especially if it was 20 generations apart. I'm pretty sure most people are aware of that. It's one reason why using flac, archiving, and EAC is so widespread.

It obviously doesn't apply to burning/copying DVD's though.

But EAC stands for Exact Audio Copy, and thus should be exact. Since it's a digital conversion process, there really is no reason why it couldn't be lossless. I've never seen a demonstration of it one way or the other, though.

The checksum comparison has been done, though. Maybe I was the peer review on that one? 

I suspect the small frequency analysis variations are simply below tolerance artifacts of CEP's frequency and statistical analysis algorithm.

But there weren't any variations. When the whole waveform was highlighted and the analysis run, it always gave the same frequency. An average, perhaps? Variations based on individual points along the waveform could be due to a variety of factors, from voltage fluctuations, temperature changes, etc., especially when you're talking about variations of thousandths of Hertz.

Thanks for all the comments!

[H₂O]

When talking CD's you are talking about bit perfect not lossless.  CD's use a basic CRC for error checking which may pass with a non- bit perfect transfer.  With md5 and even sha1 hashing collisions can occur (a situation where A hash matches even though the data does not - although the larger the hash the less likely this is to happen).  The only way to know for sure is a bit by bit comparison.

Most cd burners skip through errors (even Most dat decks skip through errors). The problem I see with CD transfers is that most discs is that most people doing the copies back in the day didn't care about quailty only quantity so alot have no lineage and where done in a very hasty fashion. 

Although CD copies are not lossy compressed many are very far from bit perfect!


Lossy/Lossless refers to compression


bit perfect refers to how accurate a copy is


lossy compression is inherently not bit perfect