Yes it’s usually worth it to try to write a really good first prompt
I can't wait until this is a solved problem because it does slow me down.
After any back and forth session I have reasonably good results asking something like "Given this workflow, how could I have prompted this better from the start to get the same results?"