To start, they were kicked out of England for being protestant, not "too fundamentalist". The schism between the Church of England and everybody else (e.g. Scottish Presbyterians) was a longstanding source of conflict (partially because the church considered itself the arm of the state, and tithes as taxes. Non-Anglicans didn't want to pay these). And, in fact, many Puritans came back (or sent support) when Oliver Cromwell seized control. It's certainly true that the Puritans didn't come seeking religious freedom, but the ability to combine their religion and government).
They were a strongly egalitarian and highly educated culture, featuring near-universal literacy both among men and women. They had a strong dedication to economic equality -- one town distributed equal-sized plots of land to it's settlers.
They had harsh laws about adultery, which were enforced against men and women relatively equally. They also had laws against "novelties" (i.e. anything new) and wasting time, so don't think this is a defense of their modernity. But it's only vaguely accurate to describe them and "the taliban" as alike. They are both cultures with deep religious drives, but those drives are different and expressed very differently.