More weight costs more energy overhead to move whatever your actual payload is, so although electric motors may be more efficient than gas, it's still true that with batteries you have to accelerate and decelerate a lot more mass in addition to the payload (merely yourself many times). That is overhead. Regenerative braking and collecting energy on the downhill make enough difference that they are worth doing, but still don't come close to erasing what it cost to get going or go up the hill.
Weight increases road wear. A lot.
Weight is harder to stop or steer in an emergency. EVs are highly computerized and so they have excellent traction control which hides this problem in normal conditions, especially when comparing any EV to any older car with only basic or even no traction control. But put two vehicles with the same traction control including the fine conrol afforded by the electric motors, in the same situation, but one merely weighs more, and the heavier one will do worse. And "emergency situation" is really all day every day driving multiplied by millions.
Weight increases tire wear. So much so that now tire rubber in the environment is a thing.
Weight increases the wear on all mechanical parts, most apparent in brakes and engine, but in an EV those are both so much better than an ICE that the effect of weight is masked, but it's still there and actually applies to almost all parts of the vehicle. You don't see it individually but it's there worsening the overall stats on repairs and lifespans. It's like the energy overhead. It's just there making everything work harder, and that costs more of essentially every resource, not just energy but wear & tear, manufacturing, engineering, materials, and performance. "performance" doesn't mean boy racer stuff, I just mean performance of it's job. If a bearing needs to support more weight, it either needs to be made bigger, or made better, or made out of better stuff, or fail sooner, or a bit of all of those. It's a systemwide overhead like with the energy.
"Mere" weight is not free, even though it's on wheels.