Is it common practice to use the most frequent words as features ? It looks like they don't carry much information, by definition. As a first naive approach, I'd rank the words by the inverse of how many categories they appear in, factor in the overall frequency, with a weight or something more clever and then take the top N.

Well, thinking more about it leads me to tf-idf and naive bayes (of course), at which point you pretty much already have a classifier. So it seems feature selection is learning in itself and defines the maximum accuracy you'll be able to reach ? This is border philosophical but I'd love to read more about these matters. Pointers welcome !