And how difficult is getting the papers submitted to top conferences without good academic backing. And have people been able to do it? Some examples would be helpful.
In my opinion, it's like asking "Can I become a concert pianist without taking piano lessons?" Well, sure; indeed, it's been done before.
But if you're going to put in the hard work, why not enroll in a Ph.D. program? They're designed precisely to train you to become an effective researcher. And they often offer funding.
Earning a PhD is hard. Becoming a strong researcher outside of a PhD program is even harder.
If you want to put in the necessary effort, then absent unusual circumstances I'd recommend earning a PhD. All sorts of infrastructure is in place to help you succeed.
The following is based on my decades of industry experience as well as recently completing a PhD in software engineering.
PhD candidates are effectively doing an apprenticeship to become proficient academic researchers. It takes smart, determined individuals anything from 3 to 10 years to graduate with a PhD. Over half who start drop out. Most candidates have one or more supervisors who guide, mentor and co-author papers.
Academic papers need to reference prior work. In most areas the credible references are mostly behind pay-walled repositories for which you either have to pay thousands to access or use your academic institutions access to them. The raises two issues: Firstly, if you are not up to speed with the state-of-the-art, then you can't tell whether your research is yielding paper worthy results. Secondly, you lack suitable referencing of prior-work.
All academically credible journals and conferences have a peer review process. Even experienced researchers get more rejections than acceptances. The big secret is that you need to cite "the right papers" in your background / prior-work section, in order for reviewers to take your submission seriously. It has been said that some reviewers flick to the biblio and if their papers are cited then they are more in favour of an accept decision.
Since you are asking your question on HN, perhaps you already are a member of IEEE or ACM. As a minimum you could peruse some of their publications or even attend a conference in an area that is of interest to you. That will give you a far clearer picture of the standard and areas of current research. Of course, such conferences are an excellent networking opportunity which will allow you to ask your question of people who are currently active in your field of interest.
When I say industry track, I mean Microsoft Research, Google Research etc. Will that provide sufficient backing to pursue research ideas?
I think you should read @impendia's comment above. If you are really driven to do relevant research then grad school improves your chances of success.
Very informative post. But I just want to point out that the specific excerpt above is largely no longer true thanks to SciHub.
Out of curiosity I looked up some key papers in my area of MBSE and was able to find them on SciHub. In my mind that amply demonstrates that a non-academic researcher should be able to access many of the recent significant work in at least that area.
I would look up papers in the field that are in the niche you want to write about and consider if it's realistic you could write something of similar quality. Writing well (which might include writing rigorous proofs), knowing about related work and putting your work in the context of past research is essential to getting published.
Also, why do you want to publish papers? Why top conferences? Why not write an article about your work or release it as software? There's other ways to let people know about any new ground you've broken.
Let me know if you have any other suggestions.
2. To create evidence of learning
3. To create credibility as an expertise both inside/outside the field
4. To monetise expertise through consulting and other works
5. To be able to write books
6. If possible to discover and invent something that would be worth millions
This is how I'm thinking? Maybe you can help me straightened my thought process a bit!
1. Because they love their subject area, and are excited about doing this work for its own sake.
That's the entire list.
If you're competing with that (and you mentioned top conferences, so rest assured you would be), then ulterior motives are only going to take you so far. Research consists of bashing a brick with your head for two years, and then being excited when it finally moves half an inch.
I love research, and I do recommend it, but I have to think that there are easier ways to achieve your stated goals.
1. With PhD you drill deep into a niche. By the time you're done, if a dozen people in the world care about your work, that's a success. Sure, you become an expert in this super-tiny niche, but the real world hardly swoons.
2. If you want to show your work so that other people read it, academic papers is not the way to go. So much so that the trend today is to encourage researchers to write easy-to-read blog posts that explain their work.
3. Expertise comes from solving problems, and that's not really what academia is about. You've mentioned ML, which is white hot right now but might not be in ten years' time. Most academic ML work competes on marginally improving on the same datasets, which rarely matter to real-world applications. If anything, what there is not nearly enough of is building quality data sets for new tasks. This is something anyone who trains themselves with decent statistics can get into.
4. Again, experts who monetise are experts that solve real-world problems. I don't know of successful consultant who are successful BECAUSE they have a PhD.
5. Writing books is a skill in its own right. I have a friend who's done a PhD and slaved on a book out of her thesis. She's very proud of it and all, but the volume we're talking here is a few dozens, bought by academic libraries. Is that your goal?
6. I feel the only people who invent something that ends up being valuable are those who apply their wits to real pain points. Most times, it's not the lone academic that launches a successful startup, it's the clever academic along with the sharp business buddy, and some seed funding. And even then, you still have to do all the legwork of any new business. A PhD, with its 4 years of being buried in the library, is hardly training for that sort of work.
I think university is brilliant and curiosity is brilliant, and going out into the world and applying what you know to problems that you see can be incredibly rewarding. It's just that all this stuff doesn't necessitate a PhD.
Yes. For a single counterexample consider Simon Peyton Jones [1].