Honestly I think it's a silly word to use for features. In math orthogonal means roughly "at right angles" or "dot product equals zero". When two things are orthogonal it means that they are unrelated so you can deal with them independently which makes things simpler.

In describing features orthogonal is basically jargon for unrelated and adds no additional clarity imo.