While HSL is dramatically better than RGB, it still has the problem of not quite mapping intuitively to our perceptions. In particular, for a fixed lightness value, the
perceived brightness still varies widely depending on the hue and saturation (100% light and saturated blue is much darker than 100% light and saturated yellow).
For fiddling with colors, I recommend the HCL model (Hue/Chroma/Lightness), which is basically the same idea as HSL except your intuitions actually work: the perceived "grayness" (saturation) and the perceived brightness of colors can be adjusted independently (almost—I'll get to this) from each other and from the hue: http://vis4.net/blog/posts/avoid-equidistant-hsv-colors/
There are downsides, of course, which can all be traced back to one root problem with HCL: it's determined by biology, not physics. This means that any implementation is only going to be an approximation, which is the "almost" I mentioned above: when you muck around with chroma and hue, you'll still see small variations in brightness. This also means that the boundaries of this color space don't form a cube like RGB, nor a cylinder or double-cone like HSL and HSV. Those spaces form super nice shapes because you can physically choose 256 values for each of the 3 components of a pixel (the RGB cube), and the others are straightforward transformations of thereof. But they have no straightforward relationship to perceived brightness nor perceived "grayness". For the same reason, there's no native support for manipulating HCL colors in browsers, and I believe it's not great on other platforms either.