Those with a PhD working in data science - do you feel like it makes a difference? Was doing the PhD worth it for you? What advice would you give to anyone considering it?
If you are using existing methods to analyze data relevant to your employer (so like trying to optimize the conversion funnel at a startup or something), PhD is unnecessary.
If you are trying to invent new statistical techniques (say deep learning researcher at DeepMind), PhD is helpful.
Also, different groups have different cultures. Some groups want to function like academia, with reading groups and paper talks and stuff. Some want to function like software companies with standups and Trello boards. A PhD will fit in better with the former type of group.
Edit: I dropped out of a PhD to go work, and that was optimal for me. I got a couple extra years to independently build my skillset, I was very hot on the recruiting market being a step above the new grads, but I didn't waste ages learning skills that would ultimately be irrelevant (published no papers.)
Another thing I've observed is that even when a PhD is required, companies usually add "or equivalent work experience" (4+ YOE) in their requirements.
That can be mitigated if you have the mathematical / statistical chops, or if you happen to work at the right place on the right problem.
So basically a PhD would be momentum for you to get on the elite track, but there are still plenty of ways to stand out. If you are the guy who designs the research infrastructure, or builds the team, and the group is wildly successful, you can still make it.
Many don't think it should even be a term at all.
This also depends on your long-term career goals. If you want to eventually manage teams or products, the years you put into a PhD might actually take away from experience around product design, strategy, management, and communication, all of which are critical to moving into leadership roles.
Some hiring managers even look down on PhDs as they can sometimes be too pedantic, nuanced, and miss the "bigger picture" around decision making. I wouldn't personally go this far but just be aware of the tradeoffs and biases.
Of course, if your only goal is to do research then you should probably get a PhD...
I've run a few webinars with people who have moved from academia to industrial data science, and here are two talks you might appreciate: https://phaseai.com/resources/from-phd-to-applied-ml and https://phaseai.com/resources/career-data-product-management
You mention "top research jobs" which gives me a reasonable vector to assume on, but I'll make some noise from this corner that there are other routes: I've managed to get my name on some reasonable bits of data-oriented/ML-based patents and publications, while still in a more "typical" career; ranging from sysop/research software support, to a SWE in big-tech doing "data-scienc-ey" work internally, but usually with direct applications as opposed to truly theoretical.
Now, I certainly worked with some PHD data scientists in those jobs, but arrogantly, I did not find myself "at a loss" in terms of providing value and pulling my weight. To your question about how I managed this, I'd attribute it half to having prepared well for this: oriented my undergrad and masters to be a split of CS and applied math, with heavy emphasis on prob/stat, graph theory, vector/discrete math, the sort of things that I got a sense were good to have on a toolbelt; and half to using these in progressively more involved "Data-oriented jobs" that didn't require upfront credentials: adminning petascale clusters, helping researchers utilize them, working in industry spaces heavily overlapping ML, data science tooling, and data/textual processing. Eventually you'll build up a track record such that the PHD isn't necessary to demonstrate that you can deliver good work in that space.
To be evenhanded, I absolutely would have trouble getting a "pure research" position as a PI, or running a lab. But I've never really wanted that, and never really regretted (actually, felt quite vindicated by) my choice to stop at a Masters degree. Again, I just want to emphasize: the most important question to me would be figuring out what YOU want, and whether a PHD is a non-starter to open those doors.
(For instance, I was told years ago it was not realistic to get a job at MSR without a PHD, except as a research-SWE, not a full scientist. This may have changed but I assume it's still roughly true; my wife has run up against similar issues in biotech where to get the "Scientist" title, even if you have the same competencies built up over time in practice, you need a PHD. This is slowly changing over time, especially as programming becomes more critical and the phd system continues to be... a bit disfunctional, but that's a slow slow change.)
If you want a research job at a large tech company or top research institute, you'll need a PhD and a history of publications. That being said, I've found there are opportunities for you be successful and work on interesting problems if you're willing to be creative and flexible.
In larger organizations my experience has been variable. When I was working at Harvard (doing institutional research which was data science for higher ed policy and strategy), my ceiling was definitely capped. There was a ridiculous amount of institutional politics and having a PhD was required for many opportunities. In contrast at Liberty Mutual (large insurance company), I found you could find your way into a data science role and follow traditional ladders for career advancement. A PhD was nice but not necessary. The downside was that your work was very confined, there was massive layers of middle management and bureaucracy, and other challenges. But if that environment was good for you, you could definitely be successful.
I also a had brief stint working as a technical project manager at a AI research institute (Allen AI). There having a PhD was definitely needed to do research. There were research engineer positions, which depending on the research group, could have you implement models, but having a phd was required to actual research.
I ultimately found my sweet spot working a early stage startups. I had the agency and autonomy of researcher at say Google Brain or Facebook, the ability to work on cutting edge problems, and the ability to get involved in other aspects of the business as well. The obvious downside is that startups are volatile (the first one I was at went down under) and while my compensation is more than comfortable (and even better than when I was at Harvard) its nowhere near the compensation of large tech companies or corporations. If you're interested in publishing, it's also a bit tricky to balance your work and carve time to do publishable research. Ultimately, I found work at at the start-up stage was perfect for me but was a bit unsustainable long term.
Ultimately, my choice to go back to grad school for my phd was for mixture of reasons. Personally, I wanted the space and time to build a strong foundation in theory (having come from an applied background), have the ability to do pure research, and ideally put myself in a situation down the line to be competitive for research position at a larger tech company or research institute. A PhD is useful really for two things, one is the credential and two is the time it provides you to build up a publication history and work on a particular problem.