I'd be fairly concerned if undersea mining waste or debris started killing off phytoplankton, perhaps by disrupting any of the delicate cycles such as phosphorus, or providing extra iron which may lead to runaway algal blooms or such, and subsequent algal death and oxygen shortage. Since the mining is so deep, we don't know if these effects will stay localized, or if undersea currents will dilute these effects, or anything.

It should be valid to question what harm there could be. Nature is quite fragile, and we should know that, especially seeing what harm industrial processes have had on the atmosphere.

On the other hand, it's quite possible that undersea mining may produce less harm than surface mining, where waste gets dliuted far enough to have less impact. Maybe similar to how salmon farming has to be done where there's a strong enough current to dilute the waste.

It's all hypothetical, but it's not unimaginable that there are nasty things laying under the layers of ocean snow. Either materials that could poison the ocean, or forms of ancient bacteria that have been dormant (like the anthrax in Siberia that used to be frozen under permafrost).

It would be catastrophic if we released something that could displace phytoplankton in the food chain, but didn't absorb CO2.

It's also possible that those minerals are important to some kind of natural process. I'm not a scientist, so I don't know what that might be, but it's not unfathomable that it's part of some bacterial lifecycle, and that disrupting that bacterial lifecycle could have implications for the rest of ocean life.