I think the cost for voting for minority extremist parties and policies should be higher, not lower, yes.
+ liberal democracies have (at least in theory) unalienable basic rights that can't be taken away, even by majority. Eg just because a 90+% majority votes to revoke citizenship from all muslims that doesn't mean that it will go through.
As for these "inalienable" right, I'm not aware of any system without a mechanism for altering their constitutional laws.
To your specific example, the UK just had a government representing a minority of voters vote through a law that allows them to strip citizenship from millions as they please.
we're not talking about some ambiguous grey zone stuff here
* proposed a drastic reduction in immigration with particular reference to immigrants from Asia.
* stated that African immigrants were bringing diseases into Australia and were of "no benefit to this country whatsoever"
* called for a ban on Muslim immigration to Australia
* announced policies including installing CCTV cameras in all existing mosques
* stated "all terrorist attacks in this country have been by Muslims"
it's safe to call the above extremist and xenophobic
A huge danger of systems that downplay the support for extremist parties is that they create an incentive for infiltrating the bigger parties instead.