Honestly getting past people isn't that much of an issue. There are normally passing spots where you pull over to let people through.
"Chelsea Tractor" is more of a dig at people Range Rovers for the looks and it never been using off-road.
There is a brand in the UK that have decided to "own" the label. Not sure why you would want/need a Ineos Grenadier in London, but some people will buy one.