Swede here. I asked my wife too, her reply is that the color "khaki" is a shade of green - "khakigrönt". But the specific style of pants - "khakibyxor" - are generally beige.
According to NE thesaurus:
"ljus, gul-brun-grå färg som påminner om (väg)damm el. sand: kakifärgad
HIST.: sedan 1904"