As a non-performance (including weight) practical matter as well, cadmium is just plain a rather toxic heavy metal. IIRC the EU has outright banned NiCd batteries entirely with a few very minimal legacy exceptions (medical devices I think?). Even though in principle part of the price is supposed to cover proper disposal EOL, people being people plenty got tossed and it's nasty stuff to have in a landfill even let alone elsewhere. There isn't any real treatment for it either as far as I know, normal heavy metal treatments like chelation therapy don't really work with Cd.
So I think yet another benefit of lithium is also that it's relatively easier to deal with over its whole life cycle. Elemental lithium is of course quite reactive, don't want to be directly exposed to it, but it doesn't stay that way in nature. That it's extremely abundant and distributed doesn't hurt either. While for economic reasons it makes sense to get it initial from naturally highly concentrated sources, and to recycle lithium packs, it'd be perfectly viable to pull it right out of the ocean with solar power if there was ever a need.