Unless there’s a massive breakthrough in battery tech, the manufacturing of sufficient batteries would be a guaranteed environmental disaster.
But since batteries can be used a few thousand times and then can be recycled efficiently, the volume required is a tiny fraction of the petroleum requirements.
Other materials are used in even smaller quantities. Cobalt is a disaster, but grid batteries don't use cobalt, they generally use nickel & iron.
It is a large environmental impact. But compared to the alternative, it's very green.
When speaking in relative terms to a newly built fossil fuel plant you are most likely correct. You'd be hard pressed to find a plausible source of energy that's worse for the environment (immediate AND long term) than burning dirty sequestered carbon sources like coal and oil.
When speaking in relative terms to a newly built nuclear plant—the context of my response—batteries would be significantly worse per KWh of baseload energy supplied.