yes - one of the reasons behind Dragonfly conception was that it's hard to write high-performant code for all architectures at the same time. Ditching ARM64 and others means much cleaner codebase and less effort to maintain all of this.
If portability of other OSes has taught us anything, it is that the interfaces are much cleaner if you are to support multiple architectures.
I think it comes down to having people/companies interested in porting it to ARM64, not about cleaner codebases.
You are absolutely correct, however DragonFly focused since the beginning on x86, that was the target platform for a cluster and the architecture every dev had. Maybe it's time to rethink that.