Good-bye ESNI, hello ECH

I feel like I'm missing something here. It seems like ECH is still vulnerable to a MITM attack, allowing the attacker to see what SNI destination the user is attempting to connect to. I'm imagining something like this:

Client sends ClientHelloOuter, with encrypted ClientHelloInner.

Malicious MITM replaces ClientHelloInner with malformed data so that decryption will fail.

Server sees that decryption fails and proceeds with the ClientHelloOuter handshake, sending back the correct public key.

Since this message is (from what I can tell) in the clear, the MITM can replace the public key with its own public key.

Client then retries with a new ClientHello, containing a ClientHelloInner encrypted with the attacker's public key.

Attacker decrypts ClientHelloInner, re-encrypts with the server's real public key, and forwards. The attacker is now able to listen in on SNI, ALPN, etc, although the HTTPS traffic itself is still protected as the attacker can't forge the certificate.

This seems like a fairly trivial attack, which is why I think I must be missing something. How does ECH prevent an attacker from swapping out the server's public key with their own, in the event that the first ClientHelloInner fails?