IMHO (15+ years of Emacs and counting) you are not wrong.
First, Emacs already has excellent LaTeX support. Unless org-mode brings you any major advantage over plain LaTeX (which I doubt it is when you're writing a paper, that's not exactly 400 pages of loosely-coupled ideas) the extra headaches of dealing with one more intermediary language and set of packages isn't useful. If hacking on org-mode is your hobby, then yes, it's an enjoyable experience. But if you're just using it, then it's one of those cases where knowing you can is not enough, and you should maybe stop for a second and think if you should.
Second, I don't think anyone ever intended for org-mode to be used (as a wrapper) for authoring scientific papers. It's great that you can export your notes to LaTeX because it allows you to easily integrate them with existing manuscripts, it allows you have formulae embedded in them and so on. It just so happens that it's so good, and so flexible, that you can do it -- but there's no world in which a simple (no matter how flexible!) export engine for/wrapper around a language will ever be able to deal with complex matters that even said language can barely deal with.
It's one thing to use it for authoring simple documents (like blog posts), that works great. But when it takes hours of fiddling just to get the editor-supplied template to play nice with your text and figures, the last thing you need is one more moving piece.