This is the most difficult part of a PhD for me.
Note: This quote is from the article on which this thread was pointing to before it has been changed by mods. https://news.ycombinator.com/item?id=13382223
Poorly defined goals should not be unfamiliar or scary to anyone who has done at least one IT project for money. Actually, I haven't encountered many well defined goals in my professional life after graduating high school many years ago.
"Ability to work with poorly defined goals AND almost no one knows about this subject you are discovering".
So of course you could start a new Web App for a customer and have poorly defined goals (good luck with that!), but the problems you might encounter have solutions already fleshed out by millions of other Web App devs (Stack Overflow). Something tells me undergoing a PhD is not quite the same, as the 'resources' in this context are a few orders of magnitude scarcer, and the subject matter a few orders of magnitude more complex ...
(I am not a PhD, please forgive me if this is wrong)
In industry goals are maybe poorly defined, but at least you have directions, and people that can answer the questions you may ask. If the contract is about creating a website, at least you know what you are going to achieve, even if you may have to fight with the customer to understand what he wants/needs on this website and what problem he is trying to solve.
Using the image of pushing the boundary: Doing a PhD is like wandering within some heavy fog. You want to push the boundary, but first you have to find the boundary. You don't know in which direction is the closest boundary as you can not see further than 10 meters from you. Maybe also the closest boundary is on top of a 8000 meters high mountain that you don't have the capacities to climb. So at the end you spend you time trying to figure out in which direction to go. Some PhD students may have supervisors that are guiding them through the fog, as they already explored the area and know it, but some other are on their own, trying to figure out a boundary to push that is accessible for them.
Situations of all PhD students are probably different, but this is my feeling about it.
I think, you rather meant poorly defined requirements for an IT project. The goals of the project do not matter for you, you get the requirements from a paid project and the one responsible of the project has to think about the goals. While during PhD you have to define your own goals.
Also, as on the picture in the post, you have to push from the circle, you have to find it where you want to push and you have to define how much should you push. While doing an IT project, you are usually moving inside the circle, i.e. you are usually combining working concepts.
In my own experience, I had a year that was almost completely wasted. I almost gave up, but instead I decided to stop trying to do work that other people wanted. I wrote the two chapters I wanted to write, because it was the only way they were going to get written, and in the end the committee was fine with them.
I'm trying to figure out if it's the right choice for me to pursue a PhD.
What was your research experience like?
I've seen a few sources mention the advisor conflict of interest (no incentive to help you finish and graduate because they will lose you after) that you mentioned earlier. Do you think this always a is a significant problem, or only if one is unaware of it?
The second time around, I think I still went into grad school for the wrong reasons, but they were different wrong reasons. After dealing with a lot of people who were woefully underprepared to work in industry, I thought getting a Ph.D. would put me in a position to fix the next generation. I underestimated how little the academy wants to do that. I opted not to continue in academia once I got my Ph.D.
Regarding my specific research experience, I'm not sure where to begin. You work like a dog, your advisor probably doesn't agree with your conclusions, might not agree with your premises either, and he wants you to go back and read an ocean of literature. Repeat for 4-10 years.
Advisor conflict of interest varies from advisor to advisor. Some of them like to produce lots of successful students, others like free labor. Either way, bear in mind that the advisor has his own agenda and is not just advising you out of the goodness of his heart.
Final piece of advice: read some "quit lit" to get a perspective on the academic job market. Read some of the critical responses to "quit lit" by academics who haven't quit. (http://www.theatlantic.com/notes/2015/09/no-one-cares-that-y..., https://chroniclevitae.com/news/216-why-so-many-academics-qu..., https://historiann.com/2015/10/07/tired-of-academic-quit-lit...). Bear in mind that if you're in an engineering field the job market isn't quite as bleak as in the sciences, or worse yet, the humanities. But it's no picnic either.