If I remember correctly (and I followed the discussions on LKML at the time, but it was a long time ago), Reiser4 was not merged because it had some "cool" features that could cause problems.

For example (again, IIRC), it allowed directories to have hard links. Al Viro was adamantly against it, showing that this potentially creates some very serious problems. In particular, it allows for cycles in the directory graph (it's no more a tree), and Viro has shown that detecting or preventing such cycles may by prohibitively expensive, and undetected they would cause any program that does directory walking (like search) to loop infinitely.

This was not the only problematic feature, just the one I remember more vividly (in part, because of Al Viro's caustic argument style :-) ).