Probably not a good idea. Really spent batteries tend to leak and damage the device.
Not working makes the owner change the batteries.
It will kill rechargeable lithium batteries.
If it's built-in lithium then you're as likely put in a step-down or a buck-boost to regulate raw lithium cell pack voltages to whatever the device needs internally, but that also needs a self-protection cutoff.
Something like the TI TPS61291[1] draws only 15nA in bypass operation, and maybe 85% efficient in boost.
[1] The bypass switch itself will have some resistance, the part below is ~1.2ohm, which might be significant depending on your load.
The same applies to my wall-clock (which uses a single battery)
Are you sure? That's pretty uncommon. Are the poles oriented in the same direction or opposite ones? Is one set of poles connected to a pair of terminals shared by a single conductor and the other set using two distinct terminals?