I appreciate and agree with your main point, but I want to push back on the "impossible to solve" assertion because solving hard problems is fun.
Using latency seems intrinsically expensive because there's no lower limit to how fast a network switch or relay can operate, meaning speed of light is the only real limit. And then the latency bounds get extremely tight, which probably means expensive components. Probably too expensive for a dedicated key fob, but maybe possible with a phone.
But position is actually the thing we care about. If the key knows its position, say with GPS, then we could do it. The key could securely sign its location (plus timestamp or nonce to avoid replay attacks) and then the car could explicitly verify the signature and that the key is within range. AFAICT this is totally secure and reasonable, if a bit expensive, to implement. In fact it seems like it would almost just work today for phone-based keys.
Putting GPS into a dedicated key fob is probably not even too expensive - car key fobs regularly cost hundreds of dollars to replace, even if their BOM is trivial, and a cheap GPS watch is approaching $100. The biggest barrier I see here is battery life on the key - neither phones nor watches like to be constantly tracking GPS because of the power draw.
I wonder what else could work. Quantum communication protocols can detect or resist relays. These are WAAY out of reach though - mostly theoretical, but IIRC the Chinese actually built a satellite to do relay-resistant quantum key distribution.
I'm not arguing that these options are things car companies are going to do any time soon. But it's fun to think about. And I don't think it's impossibly far off.