This is a big topic for debate, and will be on the agenda at KubeCon in O(days).
With CRDs, we could have each ingress controller provide its own, native ingress object ("nginx-ingress") that had the exact features it supported (with schema validation). The ingress controller would then create or delete cloud-specific CRDs ("google-loadbalancer") based on what flavour of cloud you're running under, which Kubernetes could pick up and use. Or something like that.
But as you say, some of the friction exists because cloud LBs are limited in the first place. The arbitrary cert limit on GCP is particularly egregious. We run a SaaS solution with about 100 vendor domains, which means we've been forced to use the Nginx ingress controller and terminate TLS there, instead of at the GLB level where it arguably belongs. (We could run 10 GLBs, but that would require splitting our ingresses into 10 separate ingresses, with the duplication and potential for copy/paste errors that would ensue.)
But thirdly, it's also true that several of the ingress implementations are just a bit sloppy. Traefik, Voyager and haproxy-ingress all have issues with using TLS certs (all of them have open issues about serving both HTTP and HTTPS at the same time, I believe). A lot of today's ergonomics could be solved by polishing up these projects.
But low-complexity apps don't stay that way.
What you're describing is very much the way my brain has gone. In my experience, most users end up using at least one non-portable annotation on Ingress. The logical conclusion then, is that people care about features MORE than portability in this facet of the API.
This is not surprising to me, given how religious the debate tends to be...