The base ISA is set by ARM, but every vendor has their own set of features turned on or off, and their own extensions, and potentially their own way of laying out memory mapped IO and identifying peripherals.
Every vendor does not have their own set of extensions. Different x86 chips have different extensions (from a pre-defined set) - I don't see panic about x86 fragmentation.
Is there any difference between AMD and Intel these days in terms of instructions other than AVX512?
Not aware of any but there are significant extension variations even within Intel e.g. lack of AVX on Atom.
It's not panic but this is really annoying like with AVX-512. The side effect is that people are making choice for the target architecture.