Oh, I see what you mean. You mean that for the definition of groups, topology, etc. I need to refer to a mathematical theory like set theory within first-order logic. Ok, let's do that. But then there is a definition of what "T is a topos" means, in terms of set theory based on first-order logic, right? Obviously, that definition cannot include the category of sets, because that category wouldn't be a set itself, so I am fine with considering only "small" toposes which are representable as sets, just as I am fine with just considering "small" groups, "small" topologies, and "small" categories.