I do exactly what you describe with grub2-filemanager: I keep a NTFS partition on every drive, with a few ISO files: each can be >4G (unlike with a VFAT partition like the EFI partition)
In case the UEFI boot fails, grub2-filemanager is the next entry in the boot order.
Having everything inside an ISO is easier for maintenance. I often have Ubuntu LTS and the Windows install ISO: it contains a Windows PE that can be useful for rescue (chkdsk etc)
I also leave inside a few basic scripts to add grub2-filemanager to the UEFI boot at the bottom of the list (after the regular UKIs) and a few static binaries for rescues.
I'm now moving my binaries to cosmopolitan, mostly to make it easier to have their configuration file follow: cosmopolitan binaries have a /zip part where you can stuff say bashrc, which will be automatically used by the cosmopolitan bash if compiled with the right option to check /zip for config files.
When I'm done, my "rescue" NTFS partitions should only contain a handful of files (mostly ISOs and cosmopolitan binaries)
(I would also use NTFS were I in your shoes, but that's only because I like Windows and wouldn't even use GRUB as a bootloader to begin with.)
Like you, I also hate grub and prefer UKIs, but grub2filemanager is just a list to pick ISOs like in Ventoy, so I'm fine with it!
To do the job, grub only needs to show me the list of files and read the ISO file when I select it.
There's no need for a write access, so no need for ext4 - also, if I was using ext4, I couldn't see the files or add new ones from Windows
I can use the same partition as the Windows recovery partition (where .wim files are stored) - this is a dirtier trick, but a Windows recovery partition is needed to boot Windows with Bitlocker, so why not put it to a good use and keep other files?
It's a "special" partition so I can be sure Windows won't touch them and won't easily let the partition be mounted (or formatted by accident)
Overall, it's easier for me to keep this same standard partition scheme everywhere (efi=part1, msr=part2, ntfs=part3, regular OS=part4+) as it's peanuts (even with a 1G EFI and a 10G NTFS) in comparison to the comfort it provides