Webpack and Babel are very important tools, and I'd agree that it's worth knowing how they work and how to configure them... _eventually_. However, I disagree that those are "the basics". You shouldn't have to be required to set up Webpack and Babel as a prerequisite for being able to play around with or learn React. (That may not be exactly what you were suggesting, but it's a somewhat common sentiment.)
It's worth noting that Create-React-App is the recommended tool for most people to use when starting a new React project. Also, CRA isn't a "boilerplate". It's a prepackaged build system that can be upgraded, and acts as an abstraction layer over Webpack and Babel. It doesn't come with dozens of app-level dependencies already installed and custom-configured, like most boilerplates do. It comes with sensible default settings out of the box, has a linting config that tries to detect common errors, and includes a lot of really nice bits of DX aimed at newer users.
Overall, CRA serves three primary purposes: it allows React learners to set up an environment without having to learn Webpack and Babel first; it allows experienced React devs to spin up a project without having to do all the configuration work (or copy and paste it from somewhere); and it also provides a common starting point for instructions and tutorials. For example, my blog post on using the Cesium.js 3D globe library with Webpack and React was able to start by just saying "Create a CRA project, eject, and modify these two config values".
So, my advice to most React developers would be to use CRA as the default tool for starting a new React app, unless you know ahead of time that you will need capabilities that CRA doesn't easily support, or you're very comfortable with configuring Webpack and Babel already.
All that said, if you _do_ want to know how to set up Webpack and Babel, this is a good article, and I'll be adding it to my React/Redux links list as a useful resource.