That’s because the human body can metabolise a small quantity of cyanide. Going by Wikipedia sweet apricot kernels produce ~0.3mg of cyanide when eaten, but the toxic threshold is around 100mg for acute effects.
Still, chronic cyanide exposure isn’t good for you.