Also remember that M1 is not only ARM. It's a SOC which combines ARM cores with GPU cores and AI-targeted cores. Targeting just that is a lot more work than just porting the ARM portion from a Raspberry Pi or an Android phone. I'm sure just getting a framebuffer up was a lot of new work. Fully supporting the GPU cores will be necessary for a lot of people's use cases. Once that and most of the peripherals are sorted, maybe the AI cores will be addressed.
Hence, each ARM chip/platform often needs at least a little bit of special support (or drivers) to fully work in Linux, besides the standard ARM stuff.
Apple's platform is actually very different, with much more non-standard stuff, compared to other ARM implementations. Hence, much more extensive work was required to support them, than would be typical for other ARM platforms.
Even some of the really low-level features are different. For example, Apple have their own Interrupt Controller. Supporting it (in place of ARM's standard General Interrupt Controller) was a prerequisite to even be able to boot the linux kernel. Another weird thing about this platform is the bus configuration.
Also, Power management is something that is virtually always chip-specific, and the Apple M1 has some unusual quirks on top of that.