The market is already fragmented depending on how you look at it. There are lots of custom chips for all sorts of stuff today which has their own ISA because ARM cannot be used because it imposes the requirement to implement around 1000 instructions.
With RISC-V you could end up with LESS fragmentation, no more. A lot more of the specialized chips can start using RISC-V with particular extensions rather than inventing their own ISA.
That has benefits in that one can reuse more tools and code.
I don't see a reason why RISC-V for desktop systems or high-end smart phones should get any more fragmented than ARM. There is already a standard set of extensions RV32G for these kinds of hardware.
And at least RISC-V has been designed from scratch to handle fragmentation. The base standard gives developers a way to check what extensions are supported in code. It also allows operating systems to include code to simulate unsupported instructions.
RISC-V is prepared for fragmentation in a way that ARM isn't. It is different to design for that possibility and find ways to deal with it as opposed to design around that idea that through strict rules one would avoid fragmentation ever happening.