Sexual side effects are rare and I haven't experienced any. I also get regular blood work done so I can keep track of my metrics and my doctor will let me know if anything goes awry.
So yeah, not really concerned and will definitely keep taking Finasteride.