Imperial College London has a pretty good rep, no? Even when compared to most universities in the UK?
Like many words in the UK, sometimes “college” is used in a more historical meaning
But this is semantics really, apparently the GP meant a very specific type of college but didn't communicate this and colleges that are universities under the hood aren't part of that unstated definition.
In any case, the ‘University of London’ is an almost meaningless umbrella institution. Its members function as separate universities for all practical purposes.