It is my experience that people in casual conversations automatically assume topic to be complexity theory, despite I'm in Europe.
Or maybe I will, just a bit ;) Because in fact, in casual conversations noone seem to be insiting, or in fact using adj. theoretical when talking about languages, semantics, logic, universal algebra, category theory. This is computer science after all, no need for special adjectives. It would be a bit like calling category theory a "theoretical mathematics".
Fact is, major part of the community uses term exclusively in the meaning we talk about. It could be argued if rightly and people outside may justifiably think in broader sense. Maybe better words could be found instead of theoretical, such as "analytical", and in fact I recall one of European departments reorganizing into two institutes, one for "Analytical CS".