I think you might be confusing saffron with poppies. Saffron is sufficiently expensive and widely used in cuisine it's hard to see it being popular for any psychoactive properties it might have compared to, say, nutmeg.