I agree, I tried several methods during my pet project [1], and all of them have their pros and cons. Looks like creating topics first and predicting them using LLM works the best

[1] https://eamag.me/2024/Automated-Paper-Classification