The problems are:
* People who do become professors have spent years training to become good researchers, and very little (if any) time training to become good teachers. They're kind of expected to be pick it up along the way, which may or may not work well.
* Some people in academia still insist on the traditional idea that university should teach academic research and nothing else, and resist including anything that aims to better prepare students (and especially PhD students) for working in industry jobs, even though the vast majority of them will.
* There are some fields where there really are few job prospects outside academia, and logically those fields should have very few PhD students, or at least make it very clear to students that they need to have a Plan B that is really more Plan A. But Professors need PhD and postdoc students as cheap, qualified labor. Some unscrupulous professors try to attract students by giving them an unrealistic view of their prospects in academia. They probably don't even think they're lying: see the previous point. This kind of thing is probably most common in fields like literature which combine bad job prospects with being very popular.
I like building stuff that people use, not writing papers about about how to build things that people will almost certainly never use.
So, in this end what did you do? Are you able to build product you care about now?
I've never regretted leaving academia for a single moment - that's not to say I don't have fond memories and I learned a lot there but ultimately I have always liked building stuff and ultimately that's only worth it if people get value from what you build. I knew a lot of great people in academia - but I was conscious that there was very much a game to be played (as is the case everywhere) and the game from what I could see wasn't one that interested me.
Are you sure you are not harsh on yourself? :) Maybe it was a good paper. If you are so worried, you can always ask the conference reviewers for comments; maybe they thought it was excellent.
> useless crap
You get enough "useless crap" down as knowledge and, who knows, one day you get to flying cars or the proof of Goldbach's conjecture.
NB It probably didn't help that I spent 6 years working on symbolic AI at the start of the 90's at it was becoming increasingly apparent to me (correctly as it turned out) that this fundamental approach didn't really work no matter how fascinating it was.
It probably depends on the field, also on the supervisor, and how much clout or willpower the student employs to force through their own vision. I made it very clear during my PhD that I was in principle paying a significant sum to be a PhD student (comparing my PhD salary to what I could go and earn in industry), so my tolerance for constraints was low.
Suffice to say, I dropped out of college and chased a quick buck on IT. At least on my university, it as expected that you'll get to academia, and of my friends, those who didn't went to other career path, are still waiting for a place in academia.