I think what you mean is the vm86 mode is not available in 64-bit protected mode (no hardware task switching and 64-bit IRET ignores EFLAGS.VM).
But you can jump into real mode from 64-bit protected mode by clearing PE and PG at the same time. There are a couple more chores to do (clear EFER.LME and CR4.PAE) if you want the real mode code to be able to enter 16- or 32-bit protected mode. This[1] is an example.
[1] https://github.com/tianocore/edk2/blob/master/MdePkg/Library...
Haswell?
I guess no v86 mode still makes it a dealbreaker for NTVDM type applications. Nobody wants to go back to a windows 95 style OS where a 16bit application runs unprotected and can stomp over anything and bring the machine to a halt. (For example, in win95 you could run debug.com and overwrite the first blocks of physical memory with zeros, which would immediately crash everything since that's where interrupt vectors like the IRQ timer is stored)
Would it be possible to bounce from 64bit long mode to 16bit real mode with some trusted code that bounces on to a mini kernel in 32bit protected mode that hosts v86 16bit applications, and then back again?
As far as I know it's possible to do all that and have a very-mixed OS.