I haven't taken any hallucinogens in over a decade, but my HPPD symptoms still persist to this day. I have visual snow syndrome, warping, palinopsia/tracers, and phosphenes. They're not debilitating, but extremely annoying and for a lot of people it sends them into suicidal ideation or panic.
As with everything else, YMMV. You might be perfectly fine or you might be one of the rare ones like me that acquire unexpected side effects despite the fact that many people tout these drugs as having mild or temporary effects. Choose wisely.