A dead/bad battery will often nominally recover and measure in the good voltage range after it's removed from the device, yet will very quickly drop voltage once placed under load.
Check voltage on the battery while it's still in the battery box under load. Or place some other similar load across the terminals when checking voltage. A good battery tester incorporates such a load, a voltmeter does not.