I experienced similar things growing up. For my case, it was usually colors. Each number was associated with a specific shade of color, but in my case it was less about the numbers themselves; it was more contextual. Eg. The number four represented different colors depending on whether it was describing the time of day, the number of floors on a building, or amount in currency.
I had brought this up in my youth only to be met with derision and threatened with being labeled "abnormal" by the authority figures, so I worked to suppress and hide this aspect. (South Korean society had a lot of backwards ideas in the 90s).