I just don't use rm directly, unless it's something very trivial and non-recursive. Start with ls, write out your path, execute it and check the results. If it looks fine, Ctrl+P → Ctrl+A → Alt+D → rm → Enter. Takes maybe two seconds more than just using rm, and definitely saved me more time over the years.