As someone just arriving to React - should I just learn React Router v4?
When you need routing, I suggest trying to use HTML5 pushState API. Figure out how to use it in React, and try to add routing to one of your apps.
After that, I think React Router 4 is going to be a great next thing to try. Get a feel for it and use it if you like it and if it solves your problems. I think RR4 is shaping up to be much better than previous versions.
Disclaimer: I work on the React team.
Try to insert routing only when you need it. When you're ready for a router, react-router is a great choice. Lots of people are already using it with great success, and you'll find a great community of people willing to help you out.