HARQ loop in LTE is tight = 3 ms(3ms for ENB, 3ms for UE, 2 ms for air/RRH = 8 ms in total). It means that in those 3 ms you need to decode ack nacks in PHY, run scheduling and then send the data on other side of PHY.
LTE PHY requires a lot of algorithms that doesn't fit well on x86 or general purpose HW like FFT or turbo decoding. Those usually are done in HW accelerators of baseband SOCs.
x86 lacks also complex arithmetic ISA, where wireless DSP do many complex multiplies in on cycle and complex ops are ~80% of signal processing done on DSP (not accelerators).
Second things is energy consumption, take e.g. Ceva XC4500 DSP core and a equivalent Xeon - it is magnitudes more efficient. In case of FFT or turbo it gets even worse.
Lastly, IQ data streams requires a lot of bandwidth ~1Gbps per one carrier in case of 20MHz. FDD LTE uses mostly 2 or 4 antennas in each direction, where TDD is good with 8 antennas, and this just one cell(sector). Newest ENB can handle up to 96 cells (with 2 antennas) in one box.
In theory you could run non-PHY layers of uplane in a server and use ENB as a L1 server, but still the lower MAC and scheduler needs tight latencies so server with DPDK is a must, but ARM/PowerPC/MIPS parts of the basebands SoCs are left idle (not really useful for PHY).
PDCP layer seems reasonable to run in SDN because of carrier aggregation and not so tight latency requirements, but this layer is very simple.
Of course you can run whole control plane in a cloud, which actually makes sense but this is not a great achievement ;-)