I have a masters degree, bailed on a phD, and even then I didn't realize it quite quickly enough. My MS program at Berkeley was quite flexible, and it was easy enough to get the requisite math. I dropped courses like Intellectual Property (cross listed with the law school), Product Development (cross listed with the business school), courses I was enjoying and would have done quite well in, to take classes like "advanced nonlinear optimization", which were a bad match for me.
Yes, nonlinear optimization will help you understand how neural nets work, but trust me, you don't need to take an advanced graduate class full of PhD students on the topic. If you have the prerequisites to the prerequisites, you understand math well enough to read about gradient descent on your own.
To be clear, there are people who belonged in this class, I just wasn't one of them.
For a lot of people, the sooner you realize a PhD isn't a good match for you, the better. The problem is, the sort of person who has the grades, test scores, and academic mindset to get into a top PhD program is almost certain to be slow to realize (and accept!) this. Add in the fact that PhD programs have astoundingly high attrition rates compared to professional doctoral programs (atrition at UCSF's med school is below 0.5%, whereas many PhD programs have attrition rates about 50% - yes, literally 100 times higher!) and you start to get a sense of why a lot of seemingly smart people get kind of snookered on PhD programs.
One big difference here is that people who masters out of STEM PhD programs can often still have a career in this field, perhaps a better renumerated one anyway. Though from what I've seen, getting a PhD in lit isn't necessarily a good path for someone who loves literature or theater, either.