Condenser mic diaphragms are generally non-conductive material coated on one side with a very thin conductive material. I think most of the materials used are not ferromagnetic, so I don't see damage occurring from a magnetic field short of MRI strength.
However, if the magnet is moving (most speakers I know of used fixed magnets, so this generally is not a problem), the moving magnetic field could induce a small current in the mic capsule/cables. This would affect your recording signal in some way.