I think this is closer to that than you think. Lab has three intensity vectors. One for each of the photoreceptors in your eye. Anything more than Lab is unnecessarily conplex IMO.
It’s more like the L is an intensity/brightness factor, and then the a and b values corresponding to the two dimensions of opponent color that neurons capture in the thalamus one step after the eye
So, sure, this "4th dimension" (for normals) might be as simple as "candelas" - truly orthogonal, but one does hear an awful lot about "ambient" or "candela contrastive" (a term I just made up) kinds of effects. (EDIT: e.g. in color calibration of projectors in dark rooms vs. living rooms, for example, but I'm sure there are many.) I am just one person, but it feels like candela brightness matters for color perception. So, maybe luminous intensity is not actually exactly orthogonal. Maybe this is all covered in the 1931 CIE documents, though.