I got the idea to augment with melatonin from a Slate Star Codex post. Since the author is a board certified psychiatrist, I decided to trust this info. Google should get you the original article, but here's my memory of it: melatonin is safe, little to no side effects, and should be taken by anyone as a first line attempt to fix sleep issues.
Melatonin is ridiculously safe. It basically has no detrimental effects in large doses. As a result, nutrition companies are in a war to make ever bigger over-the-counter doses because those sell better. But unfortunately while 5mg isn't toxic, it also makes me sleep 10 hrs and wake up groggy. As a result it's hard to find over the counter melatonin that is in the real dosage I want, 0.3 - 0.5 mg. However I found that Costco sells kids melatonin gummies with a 1 mg dosage, which is the smallest you can find on the market without paying extra. I just cut/bite the gummies in half to take approx 0.5mg a night. It's perfect, and I'm sorry it didn't work for you :(
The one thing that doesn't replicate for me from the SSC post is that he suggested taking melatonin 7(!) hours before sleep. I find I get tired upon taking melatonin immediately, and to take it early just screws ups my circadian rhythm.
Regarding the dream thing... Here's the weird thing: I have aphantasia when awake, and typically don't dream. But when I take melatonin I not only sleep better, but have vivid dreams that I remember upon waking. I guess I'm fortunate these don't turn into nightmares, generally. I guess I'm saying I can confirm your experience that melatonin leads to active dreaming, but maybe the nightmares are caused by something else? That is to say the melatonin is working to get you quality REM sleep, but some other interaction or issue is agitating you to make those dreams unbearable. Maybe worth talking to a doc about what can fix persistent nightmares.