Sure. Some of these I'm not 100% sure of, so I hope I don't make myself look too stupid (sorry, don't have access to the docs right now) but here goes:
- Too few registers
Mostly fixed in x86_64, but still an issue if you
want to support 99% of the machines out there still
on 32 bits
- Overlapping register classes
This makes the register allocator's job much more
difficult, since not only do registers interfere with
other registers which hold the same mode of varible,
they sometimes -- but not always -- interfere with
other registers of the same class.
- Irregular instructions
If you use some registers, they're encoded
in less bytes than others, so the optimizer/allocator
has yet another parameter to take into account.
- Fixed registers for integer multiplication/division
You're forced to clobber %eax/%edx and one other reg of
your choice. This causes more spills than it should,
and makes the job of the register allocator harder.
- Way too many jump variants
I think I counted 63 variants of the instructions,
and you can encode the same one multiple times, which
gives you hundreds of ways of encoding a jump.
- Complicated encoding
First, it's a variable length encoding. You can have
one, two, or 3 byte instruction opcodes, with various
prefixes and suffixes.
- Ambiguity in some cases
Instructions can sometimes take the same initial byte
sequences as a prefix, despite not actually having
that prefix switched on; when decoding you have to
figure out the rest of the instruction before you
decide if you've got a prefix.
- Useless Instructions (minor annoyance)
x86 has lots (and lots) of instructions that are
unused by compilers, are slower than simply doing
the smaller instructions that they are composed of,
but are still around for compatibility. Sure, it's
needed, but do I have to like it?
Really, x86 implementations these days are quite good, but the instruction set is not pretty. x86 has more cruft than most other architectures out there. It's certainly not impossible to write a good compiler for it, it's just a whole lot harder, especially when it comes to doing register allocation. (x86 isn't even too painful to write by hand!)