Actually, you'd be keeping people in prison who you know have a disregard for the law, and decide for themselves what they think is okay. They are criminals, they broke the law at the time, and as such, are people that deserve to spend time in prison.
Of course, that's a very broad way to think of it. I'm sure for this particular topic there are lots of cases of outright discrimination which means people are serving sentences for marijuana-related laws that just aren't helpful to society at all (probably detrimental).
So, you may be right regarding this particular law, however, I think it is a very dangerous precedent to simply pardon anyone who committed a crime, for which that act is now legal.
If someone believes something that's currently illegal should be made legal, there are non-anarchist ways to deal with it than simply disregarding authority and committing a crime.