This is not completely true. You can use a generic kernel with a custom device tree.

The only problem is that distributions currently tend to package them together, but that shouldn't be obligatory.