If you specify that you just want bog-standard markdown then it will not generate smart quotes. To wit:
- `pandoc foo.tex -t markdown foo.md` will not produce smart quotes.
- `pandoc foo.tex -t markdown-smart foo.md` will produce smart quotes.
- `pandoc foo.tex -t markdown foo.md` will not produce smart quotes.
- `pandoc foo.tex -t markdown-smart foo.md` will produce smart quotes.
https://pandoc.org/MANUAL.html#extension-smart
The meaning of the -smart extension on option names is inverted in some cases, and enabled by default on markdown output.