Unfortunately, the “math” of color models like HSL is only very loosely related to the science of human color perception. It was a model developed by computer graphics researchers of the 1970s who were not very knowledgeable about 20th century color science and dealing with very limited hardware capabilities compared to today, roughly based on outdated ideas from the 17th–19th century... and frankly not very well designed even within those constraints. As a result, leaning too heavily on HSL (especially for novices) typically leads to garbage results. An experienced artist picking colors visually will do much better than someone largely relying on HSL numbers.
If you want to pick colors based on numbers, look up the Munsell color system (a big lookup table from the early 20th century; the “modern” version was developed by the leading American color scientists of the 1940s), CIELAB (from the 1970s, an inferior but based-on-a-simple-formula Munsell clone), or CIECAM or IPT models (more recent and more complicated formulas).