Alkaline button batteries used to contain mercury, which has been phased out in the last 20-25 years. The original Voyager line used 3 x LR44 (alkaline), but even then you could have replaced them with their less toxic silver-oxide equivalents. Today's LR44 cells are mercury-free.
Later iterations of HP12C use 2 x CR2032, not because of any concerns about toxicity -- the 2032s were equally toxic -- but, I suspect, because the original battery controller was all custom HP silicon in some funky semi-analog process, which couldn't survive when HP, along with the rest of the industry, fully succumbed to cost-cutting disease. Everything possible in the new calculators if off-the-shelf.