It's hard to make sweeping generalizations about its historical use but it's safe to say that the psychoactive contents of ancient strains of cannabis was much lower than today's and its use was, while not exclusively so, often restricted to rituals and people for whom the psychoactive effects were socially important (shamans etc.).
More recently, up through the 19th century, cannabis was recreationally used by plenty of people with no legal preventatives and it certainly doesn't seem like everyone were "potheads". But many of them weren't white, which, to the legal minds of the time, was just about as bad.