You could provide exceptions for a primary residence where the increased taxes would be sufficiently burdensome on the owner's income. Doesn't have to be all or nothing.
Just don't do what Prop 13 did in California where they applied the property tax limit even to commercial property for some reason.