I get the point he is trying to make, but my problem with this argument is that it seems to be based on an idea that the default is that a thing is illegal until a law explicitly allows it, which is contrary to the way I've always understood things to be.

In other words, in some (perhaps distant) past, marijuana was not illegal until a legislator decided to pass a law prohibiting its use.