My wife was fatally injured in her Ph.D. program. The OP outlines a lot of just what happened to her. To have time to try to help her, for a while I took a slot as a B-school prof. It didn't work -- lost her anyway.
I never for even a milli, micro, nano, pico, femto second wanted to be a prof. Instead, I wanted to be solving problems in business, the money making kind.
(5) Non-Academic Career. Then I tried to get my career going again, outside academics. Bluntly, that didn't work very well.
I made a mistake: I should have returned to DC and gotten back into applied math and computing for US national security. I guessed that there would be opportunities as an employee in business; I was wrong.
Bluntly, my view is that US business and Ph.D. holders mix less well than oil and water.
Part of why:
(A) Business is still a lot like Ford in Henry's day: The manager knows more, and the subordinate is there to add muscle to the work of the manager. A manager has no use for a subordinate who knows much and resents or feels threatened by such a person.
Supposedly lawyers have a solution: A working level lawyer should work only for a lawyer. Period.
Well, a working level Ph.D. should work only for another Ph.D., and that criterion would eliminate nearly all jobs for a Ph.D. in business.
Not even a CEO wants a Ph.D. around except maybe tucked away in some side organization, out of the main work of the business. E.g., the CEO is plenty sure that he is the only really important person in the company and, thus, certainly doesn't need a Ph.D. or some academic background he (the CEO) doesn't have!
(B) Business regards Ph.D. holders as blue sky dreamers out in the ozone who refuse to contribute to the business, who really want to publish a lot of papers and get a prof slot in academics.
(C) If a Ph.D. person does anything original relevant to anything in business, usually the business will regard this person as a threat.
(D) Suppose a Ph.D. takes on a practical business problem:
(i) If the Ph.D. successfully uses their advanced knowledge to get a good solution, e.g., one that makes a lot of money for the business, then everyone else in the business, even the CEO and the BoD, will feel threatened and/or jealous.
(ii) If the Ph.D. fails to get a good solution, then everyone else will take the opportunity to denigrate both the person and the Ph.D. degree -- "I always thought that a Ph.D. was just a useless, hopeless, worthless impractical dreamer out in the ozone, and now we know for sure.".
(6) A Ph.D. in a business research division. Yes, some businesses, say, ones with some loose cash, might set up a research division, hire a Ph.D. as the director, and hope for something good. If nothing good happens, well, the company could afford the wasted money.
Generally, connections about the actual business between the research division and the rest of the company are more awkward than a skunk at a Victorian garden party. The rest of the company doesn't want to be bothered, sees various threats, etc.
Here are some of the reasons for such a research division:
(A) Luster. Use the research division to impress the public, for good PR, to impress customers, to cover the rear exhaust port of both the CEO and the BoD, etc.
(B) As a patent shop. So, the research division can develop a patent portfolio, maybe dozens, hundreds, thousands of patents. Then some specialized lawyers can use that patent portfolio as a, call it, battering ram against any would be competitors. There can be cross licensing deals, revenue, etc.
(7) Career direction. It's your career. In this career, there will necessarily be some directions you will be pursuing. Some directions are good; most are not. It's up to you, and maybe your family, closest, trusted friends, etc. to pick, at least try to pick, a good direction(s).
If you just look for a job, get some offers, and take the best offer, then likely you will be following the direction of your employer, especially your immediate supervisor. That direction was not picked by you; likely it is not a very good direction for you or anyone; likely in that job you will have quite limited opportunities to change the direction to be something good for you.
Bluntly, you will want income enough to provide for food, clothing, shelter, transportation, medical care, insurance against risk, recreation, a house you own, a family, education and other needs for your kids, and retirement, with some security, i.e., low risk, and at least a comfortable life style. That obvious goal is surprisingly difficult to achieve, especially if you are working just for a salary for a manager in a company, small, medium, or large.
(8) Blunt US Fact of Life. IMHO, nearly all the people in the US doing well supporting a family get their money from owning part or all of a business that makes the money needed to pay the bills for that family.
For this, can use some strategy: E.g., run the most popular Italian restaurant in a radius of 50 miles. Then you have:
(A) A strong geographical barrier to entry, that is, a restaurant more than 50 miles away will be little or no competition for you. You have a better "Buffett moat" than any of IBM, Cisco, Intel, etc.
(B) Your business is unlikely to be killed off by changes in technology.
(C) We can be sure lots of people will still want a good Italian restaurant 10, 50, 100 years from now. You have a business more stable than any of IBM, Cisco, Microsoft, Facebook, Google, Intel, etc. Good economy or poor, people will still want to go for a dinner at an Italian restaurant -- you are relatively immune from changes in the economy. You have a very wide variety of customers, i.e., are not vulnerable to some one or few customers going broke, leaving town, etc.
(D) Your family, spouse, children, can help in the restaurant and learn the business and continue running it as you grow old.
(E) Working as an employee, you can be fired by a manager who, for whatever reason, doesn't like you. If you are the owner, then you can't be fired.
(F) No one can please all the people all the time, and some managers can never be pleased. But in a good Italian restaurant, one unhappy customer occasionally can usually be mollified by an apology, a free glass of wine, just tearing up the check, etc. You DO have to do good work and please nearly everyone nearly all the time, but you can't be run out of business by just one unhappy customer.
All or nearly all of (A)-(F) apply with no more than small modifications to a huge range of Main Street US family businesses. In your career, you should aim to do at least that well.
(9) Ph.D. Entrepreneur. Okay, you have a STEM field Ph.D. and want to own your own business. If you work hard and smart, find that your Ph.D. is a great technological advantage (e.g., you can stir up powerful, valuable, new secret sauce), have some good luck, avoid too much bad luck, get well informed, consider strategy, ..., etc. then you might do really well. Your Ph.D. could be a terrific advantage.
(10) Warning. Generally, if want to use your Ph.D. to help you be an entrepreneur in something relatively new, i.e., not an Italian restaurant, then likely you need to be darned careful and insightful.
In this sense, I will say:
(A) I believe strongly in the potential of some original applied math based on some powerful pure math prerequisites.
(B) I regard current work in artificial intelligence (AI) and machine learning (ML) as not very promising. Some people may yet have good careers there, or quickly get rich from some stock, invest the money in an index fund, and essentially retire, but generally my view is that the math is not powerful enough to be very promising and 90+% of what is being done in those fields now is based on wild, blue sky dreams with little real hope and a lot of hype, PR, maybe patent games, etc.
Why: So far too much of the AI/ML work is too close to empirical curve fitting.
(i) For small amounts of data, we've been able to do, and often have done, such curve fitting going back decades to the first transistor computers. At one point in my career, inside GE I did a lot of consulting for that work. So there was, and still are, SPSS, SAS, Matlab, R, etc. I never saw such people in yacht clubs.
(ii) What appears to be new is curve fitting for large amounts of data. Well, we don't expect to have a lot of such data collections and promising corresponding problems.
For more, I'd guess that self-driving cars are not very promising: For now, for current traffic on current roads, driving occasionally, and too often, needs real human intelligence. E.g., chimpanzee intelligence is not enough, and AI/ML are a long way short of chimpanzee intelligence. There is a chance for self-driving cars on roads that have a lot of new engineering, but that will be very expensive, IMHO, for a long time, too expensive. Self driving might work on some large farms, in a big open pit copper mine, some military tasks, and some other situations much less challenging than Manhattan traffic, I-95, etc.
The general lesson: One of the keys to success is good initial problem selection. Most of the problems people have selected are not good. So, we have to try quite hard to select a good problem.
Your mileage will likely vary widely.