That is because we also can't actually directly perceive the yellow from that lemon, i.e. we don't possess cones that have their max sensitivity at 570 nm (yellow). Instead yellow is created in our brain by combining the data from the M and L cones: If both signal at about equal intensity, our brain calculates that to be yellow. So the perceived yellow can actually be 570 nm, or 540 nm (yellow-green) plus 600 nm (orange) or similar. Only if the distance between both wavelengths is too high this stops working.