I don't completely agree. A contrarian, by definition, is responsive to the political context they are in. In Chesterton's time, there was a wave of anarchism finding purchase in academic and philosophical circles: hence he appeared as a conservative.

A "conservative" from Chesterton's time doesn't necessarily map cleanly onto the modern notion of the same name... Supposing the allegorical "fence" to be something like clean air or water regulation, one could argue that it's modern "conservatives" more often than not arguing to disturb things without considering second-order effects.