It's not hard to acquire something with an ARM CPU on it, but at prices an ordinary person can afford, they're all in the category of toy computers. Try to find an affordable ARM system with a M.2 2280 NVME SSD slot on it like you can find on a $100 desktop x86-64 motherboard. Or multiple PCI-Express 3.0 x8/x16 slots.
I've previously spent many years working for a hardware manufacturer. Personal theory is that this is a real example of a chicken or egg problem related to economies of scale. Nobody wants to spend dozens of millions of dollars tooling up to produce ARM socketed CPUs and motherboards and stuff, which may or may not be price/performance competitive with current-gen Intel and AMD stuff by the time it's ready for release. And there's a huge risk in manufacturing something like that and then discovering that the sales volumes are really low.
Look at the sheer massive quantities that the top-ten Taiwanese motherboard manufacturers churn out every year.
> You can wipe pretty much any Android device (not just phones, but also HDMI "streaming boxes", which are a convenient form-factor for basing a workstation on) and install Linux on them
No you really can't, and phones aren't servers. Take any modern $600 smartphone and try installing something very close to a stock debian or centos on it. Being able to maybe boot a Linux kernel on something doesn't mean that there's anything like the market demand for that particular hardware platform target for a whole distribution.