I'm also guilty of sometimes spraying into the jack directly, and have not suffered any apparent negative consequences. I don't always do that, but will if I feel the contacts are especially dirty or if the jack needs additional cleaning. Either way, I do feel the mechanical action of the plug scrubbing against its actual contact points on the jack concentrates the cleaning action where it matters most.
The actual mating plug, or an alternate identical one which fits the jack is the idea tool for cleaning the jack. Best not to stick other things into the jack such as a skewer, matchstick, Q-tip, etc, as those things might snag and leave behind foreign material, not fully engage with and clean the actual contact areas, or strain the contact springs. I'm more concerned about something like that than the repercussions of spraying a relatively benign cleaner such as isopropyl alcohol or De-Oxit directly in there.