Why is this nonsensical? Gambling is not necessarily inter-state?
The Controlled Substances Act also rests on the interstate commerce clause. Can you imagine if the federal government was allowed to regulate heroin sales but not ban it?
[1] Wickard v. Filburn ruled that growing wheat FOR USE ON YOUR OWN FARM can be regulated as interstate commerce because otherwise that demand would tend to raise prices on a interstate grain market.