According to searchirc.com QuakeNet peaks at about 170k users, whereas Justin.TV chat has gone up to 188k concurrent users so far.
I should write something about how JTV Chat was implemented - there's some interesting stories.
You mean channels.
#trilug (Triangle Linux User's Group)
#trijug (Triangle Java User's Group)
#trilug-mtb (people in trilug who ride MTBs)
#fedora
#centos
#asterisk
#centos
#openmoko
#java
#eclipse
#spring
#jboss
#hibernate
#ruby
#erlang
#prolog
#lisp
#ai
#swig
#machinelearning
#openqabal
Plus a few bits and pieces on freenode.
Rizon: #darkhold, #cccp
Also you might want to try building from svn, I've got some builds up on http://janeylicious.com/colloquy/ if you're interested.
Also if Colloquy isn't your thing, there's also LimeChat and Linkinus. I've got lots of bitching that Colloquy's a dead project (with commits from a day ago...) and people who decide to switch have found those two clients to be useful, although Linkinus is shareware.
Most active in #django, can be a pretty fun channel at times
UnderNet: #philosophy, #trivia