It's not really an issue on "desktop class" devices you want to run Linux on and similar to x86 in practice. When people say "modern ARM devices" they are always talking about 64-bit ARMv8. Of those available chips, there simply aren't many differences in terms of available ISA features. All of the "real" differences come down to microarchitectural ones e.g. Cortex-A53 vs Cortex-A55 or whatever. But Linux distros don't need to maintain builds for all of those uarchs, anymore than Debian needs to maintain separate builds for "Haswell vs Skylake vs Zen2" on x86. You can of course compile from source and make this distinction yourself. But it's basically par for the course, like it has been in the x86 world for a while.
The reasons most ARM devices require special "builds" for $YOUR_FAVORITE_OS almost entirely comes down to bootloader/peripheral nonsense. There wasn't (until recently) any standardized way to discover devices attached to your favorite ARM chip, nor any standard boot protocol, like we have in the x86 world. Therefore every device needed its own little description of the attached peripherals and (often) needed a little support in the bootloader -- uboot -- to accompany it. Therefore there are normally device-specific kernel/bootloader packages in your favorite distro, and specific installation instructions, but not every package needs this treatment. Once you install the distro, you can use the same ARM userspace binaries/packages as any other device, more or less.
However, ARM is now settling on ACPI/UEFI like in the x86 world to handle booting and peripheral discovery in a generic manner. You can, for example, use UEFI/TianoCore on the Raspberry Pi 4 today, and boot generic ARMv8 distro images of your choosing, just like you do with x86 systems.