I have a very similar process. I start of with an org document, do a couple of drafts and then pandoc to LaTeX, pull in the proper template and add the frills as you say.