is that a detrimental effect?
It's not going to affect you like smoking cigarettes, or how drinking can hit the liver. But it definitely has some effect on ones thinking. Is this detrimental? I think that's a more subjective question.
Personally though, I consider Psychedelics to be pretty safe unless one starts playing games with high frequency usage, or high dosage. There are also outliers, for example Brian Wilson ended up having a pretty bad time with LSD.