From what I've read, it matters a great deal what time period you're talking about. Like the Ummayyads were pretty tolerant, but the Almohads were quite oppressive.