I've used both, though not for anything especially complicated. Your guess about the polish level is correct; ESXi blows Proxmox out of the water interface-wise, especially for some specific tasks like configuring networks or managing PCI passthrough devices. As far as "more limited" goes, I can't say I noticed much of a feature discrepancy, although being open and just sitting atop KVM, Proxmox doesn't have any of the artificial limitations imposed by VMware (like a cap on number of vCPUs in the free version, eugh).

One pretty big one, if you're looking at homelab use, is that Proxmox is just Debian, so it's waaaay more likely to run on crap-tier commodity hardware, whereas VMware has all the usual enterprise-level support commitments and so you're very much on your own if you try to force it to run on hardware that isn't on their official compatibility list.