You're right. Addiction can happen with anything, it just may be less common with LSD or certain other substances. I have taken LSD and other tryptamines 100s of times and at times my intake patterns probably resembled an addiction.
There is a withdrawal in a different sense I think. I had HPPD for around 3 months after heavy psychedelic use with LSD as the centerpiece of that. I was doing up to 10 tabs at once and using mushrooms and certain research chemicals every two or three days for a few months.
Not to excuse my risky behavior, but other people go much harder than I do. MDMA is a favorite choice, especially in Europe where it is cheaper. There are endless stories on Erowid and Bluelight of young people buying 10 grams of a research chemical and plowing through it in a summer. Often it's stuff like MDPV or 25i NBOME, with a short history of human use and carrying cardiac risks.
A wide range of psychedelic drugs have addictive potential IMO. I'm not sure how meaningful the distinction really is between psychological and physiological in the context of addiction.