I don't really understand what you did, but I don't know some terminology that others use here.
This is what I do, and remember that I can only do it in spots that have no singing. -
I have say a 3 minute recording of a song with a short instrumental break which someone talked over.
I use WavePad to cut the song into pieces (lots of programs can do this. I make three files - A (the beginning of the song), b (the instrumental break with unwanted talk), and C (the rest of the song).
Next I run file b through UVR5, which splits it into b-vocal and b-instrumental. I give a listen to b-vocal to make sure it includes no music that I wanted to keep (quality check).
If it's ok, I then open up b-instrumental in WavePad. I use its merge feature to combine files A, b-instrumental and C. Listen back to see how it sounds which is normally good.
I very rarely have had any problem where the lack of quiet ambient audience noise makes it sound weird.
But with my master recordings I'm already attempting to get as little of that as possible. I use only cardioid microphones, and I am always as close to the stage and the PA as I can get. I'm also usually at small music clubs where not many people in the front are talking over the singing, and usually it's so loud where I am that no talking from anyone behind me comes through, especially with my cardioid mics.