PDFs are notoriously hard to convert to anything else. Imagine a web page where every single character is positioned absolutely relative to the `body` element: This is how PDF layout works.
It's easier when the PDF also contains HTML tags. So your approach may actually work though it's kinda silly.
Pandoc[1] is a better solution.