The reasons you cite are why it is a controlled substance, and why it should be taken under supervision of a doctor. But they aren't arguments against the drug itself or that it is in a different "category" of drug. It is a treatment, and like any treatment, the pros/cons should be discussed with medical professionals and appropriate decisions made for each individual.
And yes, sadly our society fails people who are different and our medical system fails people with regards to their medication all the time - that can happen with any medication though. And yes, consequences vary greatly depending on the medication.
People with ADHD are often amused by referring to amphetamines as a "hard drug" since it has practically none of the typical hard drug effects on them.