4-5 times a month would tend to be weekends only. If it was drinking you would consider that to be pretty casual.
Every weekend is "regular" usage. I equate casual with occasional. 0-2 times month, with no regular pattern.
Fair enough, "very casual" wasn't the best adverb to use. I agree with dyladan's point though; drinking on the weekends would certainly be considered casual in my circle of friends.