The way you tilt the odds in your favor for getting a good PI-student match is to be embedded in the field. If you're an undergrad, getting involved in research is a great low-risk way to do this. People in research have reputations in their field, and people in the know will direct promising students away from labs with bad cultures.
1. Don’t opt for a phd just because it sounds cool (remember the first tweet shared in that article).
2. Academia is a pretty hard space if you can’t get your own funding or if you don’t like hustling for it. Getting tenure track jobs without having funding attached is impossible.
3. Very few jobs in industry need a PhD. Most are ok with a masters. Those that require a PhD can be selective based on where you graduate from and what your publication record is.
4. Know your PI well and reach out to his/her previous students who have already graduated. Some may reply - especially those with a bad experience.
5. Back to point #1, don’t take up the PhD just because you get the assistance and stipend. Make sure you really like your subject - it’s the only thing you’re doing for the next 4-12 years depending on your discipline while subsisting on poverty level stipends
This varies dramatically by field.
Even if you get a PhD your working career is going to be around 35 years. Add 5-10 if you're done after college, and even more if you're not doing college.
I think we need to stop pretending people are senior when they're 10-15% of the way through their career.
They probably don't have high concentrations absolutely (e.g. they're not going to be 90%+ PhDs) but it's probably still several orders of magnitude above your average industry role.
The r&d space is really interesting but very selective for obvious reasons. Just a PhD won’t get you in. You will need significant other contributions - eg the right publications in high impact journals, possibly internship with the companies themselves, the right advisor/PI that collaborates with the company, etc.
Whether it becomes a practical requirement, or formal requirement, would depend. If it's a high paid position, in a small research group, of an established field, then Ph.D.s are likely to be very important.
If its a position in a large research group in a new and fast changing field, then its likely to be more open to anyone with unusual or interesting promise, but still highly value Ph.D's.
As for fields: research in engineering, finance, economics, device physics, chemistry, biology, pharmaceuticals, etc. Anywhere the research involves highly technical knowledge, and a pre-existing track record of original thinking, personal initiative, that have produced objective results, are very important.
"Research" is a thought and project management meta/leadership skill, on top of being highly educated and talented in an area. Ph.D.'s provide the extra time and social context to fill out those skills and demonstrate a track record.
Deep research areas in computer science would be foundations of databases, cryptography, formal verification, statistical heuristics, core algorithms behind deep learning, ...
Since so many areas in computer science are relatively new, there is generally more flexibility in who might be considered great at research. The field includes many unusually creative individual practitioners who have produced great work without a Ph.D.
But Ph.D.s are still very much in demand and common for research positions in these highly technical, high value areas.
If a PI isn't meeting with each PhD student for at least 30-60 minutes per week, especially in the first few years, they are probably not doing a great job.
I always encourage students to email the current and former students of the professors they want to work with once admitted to see if they can have a video chat to discuss what it is like working in the lab. Even then, it can be hard to get frank feedback.