I don't know how feasible it is to scale that up to a datacenter though, and I expect MacOS licensing costs would wipe away any power efficiency gains. But I do wonder if they could hypothetically scale up and beat the best Intel/AMD have to offer just using Rosetta 2 to translate x86 code.
What's in it for Apple is money and better economies of scale for chips. But I don't really think it fits Apple's MO so I doubt they'll do it.
More specifically, people are running Linux on the CPu cores.
The M1 is a system-on-chip, and according to the floorplan [1], the CPUs are maybe 1/5th of the chip. There are many other features that aren't unlocked, such as GPU (which is a brand new architecture) or power management. The latter is key to exploiting the chip to its full performance envelope.
I don't expect Asahi to get anywhere further than proof-of-concept before it becomes obsolete by the march of the silicon industry.
The GPU stuff is new, but it seems like good progress is being made: https://rosenzweig.io/blog/asahi-gpu-part-3.html
For data centers, it helps that GPU and so on is just less important. It's wasted Silicon, but the CPU is competitive even before considering the added features so that's not the end of the world. There's a good chance that Apple can use that to their advantage too, by using chips with broken GPUs/NNPUs in the DC... or designing a smaller chip for data centers... or one with more cores... or so on.
As for what's in it for Apple, it would be the profit from selling a hopefully large number of chips, but adding official Linux support and also commuting to an entire new market for a minimum of three years is probably far higher a cost on focus than any potential profits.
And then they could resell the box they designed as a server, either with Linux support out of the box (unlikely, but since in this mystical scenario they'd have to write kernel patches for Linux to get it to boot...) or with a build of macOS that could be set up headlessly, in order to recoup the development costs. Apple shops that want machines to run Compressor or Xcode's build server would eat that up.
Then the gains of scale, making a CPU just of the Mac Pro would mean too low production numbers, but with data center usage would drive those up - especially if Apple also sold it to other customers, e.g. bringing the Xserve back. For the OS they could run Linux virtualized or they give the native Linux on Mac developers a hand.
What's interesting to me is to see if they'll use M-series chips in their own datacenters. They already run Linux there apparently.
Or do you mean that they'll start selling parts (SoCs)? Not in a million years :-)
AR/VR/more wearables would be another.
Home appliances/electronics would be another.
More likely I would see Apple making an exclusive server chip licensing arrangement with ARM or something similar.