Besides having 3 primary kinds of photoreceptors, there are additional complications caused by the initial processing of the color information in the visual system.
Besides the 3 primary color sensations from the red, green & blue receptors, there are a few dedicated detectors for some colors.
Yellow is detected when red is equal to green and both have a high enough brightness. White is detected when red, green & blue are equal and all have a high enough brightness. While black could be detected later, by the lack of information on the other channels, it seems that it also has a dedicated detector.
This gives 6 primary color sensations, where the colors are not perceived as a mixture of colors, like it is the case for orange, blue-green, violet, purple or gray. Presumably these 6 colors correspond to separate outputs of the initial color processing stage of the visual system.
I do not know whether this is true, but I believe that there also exists a dedicated detector for the color brown. While brown is just orange with low brightness, the "brown" sensation is very distinct and very unlike the differences perceived between e.g. dark red and light red, dark green and light green or dark blue and light blue, where the differences are perceived to be only in quantity, not in quality.
A dedicated detector for brown makes sense, because important things in the environment are brown, e.g. the ground is brown in most cases, because it is composed of a mixture of white oxides with red, yellow and black oxides of iron and manganese. Also wood is frequently brown and also many mammals are brown.