The Supreme court decided that the Fed could regular intrastate commerce in Wickard v. Filburn, and that ruling has also been used since to go after marijuana that stayed intrastate. [1]

I think it's a bullshit interpretation of the text of the Constitution too, but that doesn't change the current legal landscape, however much I or anyone else might want it to.

1. https://en.wikipedia.org/wiki/Wickard_v._Filburn#Subsequent_...