That is because performance isn't the only metric. Things like cost make a big difference as they would need to make it decently priced for silicon board partners to buy. Just the amount of L2 cache in the M1 is 3x-6x as compared to the X3 core which adds cost to the chip.
The M1/M2 is an amazing chip, however you need to consider that these chips are in different price brackets hence the performance discrepancy.