Bollocks. For starters, Brexit had support on the far-end of the left, who see the EU too liberal, e.g. https://www.ft.com/content/692f2578-fcbd-11e5-b5f5-070dca6d0... (RMT comes to mind since I work in that sector)
I've certainly seen many outside the Momentum bubble being labelled 'Tory' (e.g. Lib Dems as 'yellow Tories', Blair/Starmer as 'red Tories', etc.), but you're speaking pure hyperbole.
"Extreme far right" is reserved for the likes of NationalFront/BNP/UKIP/BritainFirst/BrexitParty/Reform/whatever they're calling themselves these days (plus their goons like EDL, DFLA, People's Front of Judea, Judean People's Front, etc.)
It saddens me to see this sort of word-muddying (especially on HN), since it makes it easier to deflect this sort of crap:
https://en.wikipedia.org/wiki/Racism_in_the_UK_Conservative_...
https://en.wikipedia.org/wiki/Islamophobia_in_the_UK_Conserv...
https://en.wikipedia.org/wiki/Antisemitism_in_the_UK_Conserv...
(Not forgetting https://en.wikipedia.org/wiki/Antisemitism_in_the_UK_Labour_... to avoid the knee-jerks; although that's probably not dismissed as 'people call everything "extreme far right" these days')