I agree that this is extremely valuable, but it's worth flagging that it's harder to reason about the impacts of class imbalance for generative models than e.g. classifiers. For example, should we think about genre imbalance per novel, per token, or on some more complex basis? Which genres are most relevant to a target distribution of chatbot queries?

This isn't to suggest that organizations shouldn't invest in actively understanding their training data, but that post-hoc bias analysis is going to be a critical component of evaluation for the foreseeable future.