Talk during the music when you feel the need / inclination as long as doing so is generally acceptible in the social situation (a bar more so than a theater, both far more than a classical performance hall where even wispering is plain bad form.. just wait for the breaks). But even when it's socially cool, do so as if the people around you to whom you are not talking were stealth recordists, even if you have no reason to think they would be. That's simply showing courtesy to everyone in the place who would rather listen cleary to the music without interspersed distracting bits of your conversation, and that's likely everyone there incuding those talking too loudly themselves, except for the person you are addressing.. and sometimes even they may feel the same way!
Sure, it's all shades of grey and is situation dependent.. but remember that music is an auditory artform, and in a good many situations at least, people show up specifically to listen to the music. Granted in some situations the reason most people are there may be as much or more for social coctail partly aspects as the music. But if the situation is mostly about the musical perfomance, please do your best not to impose too much on those around you who are focusing on it and enjoying it.
People would freak out if a stranger pulled out a flashlight to read a book next to them during the projection of a film in a theater. How is talking incessantly over the music at a serious music performance any different? Somehow that social situation is thought of quite differently, but really shouldn't be. Both compete directly with the art form by polluting it's basic medium and lowering the signal-to-noise ratio. The social scene at a well attended midnight cult-film is more like a cocktail party atmosphere, and there it's probably cool to read all you want there while people talk about the film and interact in the audience.