Specifically there is a reference counting optimization on the M1 that dramatically helps performance of compiled Swift apps - something only worthwhile if you know the majority of what the chip will ever do is run swift apps.
That’s one further reason their system is faster: they designed a system, and the design of their CPU, GPU, etc. was driven by what the final system needed.
Other system manufacturers buy individual parts, where the manufacturer of each part extends/optimizes it with only a vague knowledge of what the system it will be used in will look like (and they don’t want to focus on one specific system, as that would mean they can sell it to fewer device manufacturers)
The M1 is strongly related to their A14, which runs phones and tablets.
Also: what's between a server and a desktop machine?