There are places where jitter matters and places where jitter does not. Jitter is a clocking error. All of your samples are a 1 or a 0. Since 0 is a null, we use -1 to represent it. so all samples should be a square wave that is +1 or -1. The problem is that square waves are hard to make so the sample will actually have a slope to it rather than being vertical. Jitter is the clocking error that causes this to happen, or it is the problem reading this imperfect sample because the sample is read at a place other than the +1 or -1 location. For example, if the sample reading takes place at -.3, is this a 1 or a 0?
Now, jitter happens in the analog transfer stage and this is the only place it matters. Jitter in the AD causes a blurring of the image and loss of depth. Once you have jitter there, even with reclocking etc, it isnt going away. Must get a better clock for the AD stage. Jitter also happens in the interface and cables between 2 digital devices, but because it will always be regenerated, this has little to no effect on the sound. Digital clones dont have to worry about jitter. Finally, jitter happens after the read on the way to the DAC. This stage is also important and causes the same problems as on the way in. Getting a better clock, making the clock you have more stable with blu tack on the crystal or through more robust power supplies or better upstream AC are all ways to combat this.
Jitter should never be bad enough to cause actual distortion or dropouts, that much jitter should be easy to fix as you have done something major wrong, like running a 400ft mic cable instead of a 3 ft aes/ebu cable.
Hope this helps the education
Frost