Tesla tried that in their original Roadster. It had a two-speed transmission. The lurch when the transmission shifted during launch was too hard on the transmission, and many of them broke under warranty. Tesla had to replace many transmissions. It was easier to fix the problem on the electric side than the gear side.
There's something to be said for a paddle shifter, but mostly for better control at low speed. For parking and off-road use, it can be helpful to map more pedal angle to a range of low speeds. This is more like the high/low range lever found on off-road vehicles and tractors than like a stick shift.
Mode switches are undesirable because they lead to "wrong mode" accidents. This is an ongoing problem with aircraft automation.
Behavior with zero accelerator and brake is an interesting question. Should the vehicle creep slowly forward? That's a classic feature, or bug, of automatic transmissions. Some automatic transmissions had anti-creep mechanisms to prevent this. If the vehicle comes to a full stop, should releasing the brake initiate creep? Should the vehicle roll back if on a hill? Some cars have "hill holders" to prevent that. For an electric car, those are all design decisions - the power train has no inherent behavior.