Oh, it does matter. I've made a living at times from non-fiction storytelling, and if you don't get in the habit of sticking to the facts, it's shamefully easy to slide ever closer to George Santos territory.
But there's a compromise that will keep us both happy. Nothing wrong with having this go into the "legends" category -- where it's harmless fun to keep them circulating. Just as long as we know that this isn't quite how everything works.