If you were an NLP researcher at a university whose past years of experience is facing existential threat due to this rapid innovation causing your area to become obsolete, what would be some good areas to pivot to or refocus on?
Why the hell stay in in academia? This is clearly the next technological wave, and you shouldn't sleep on it. Especially when you're so well positioned to take advantage of your experience. You can make $500,000/yr (maybe more with all the new startups and options) and be on the bleeding edge.
If you want to go back to academia later, you can comfortably do so. Most don't, but that doesn't mean it isn't an option.
ETA: And though it may take longer, people who understand these models will eventually be in possession of the most valuable skill there is. Perhaps one of the last valuable human skills, if things go a certain direction.
Getting your hands dirty is the best way to understand how something works. Think about all the useless SE and PL work that gets done by folks who never programmed for a living, and how often faculty members in those fields with 10 yoe in industry spend their first few years back in academia just slamming ball after ball way out of the park.
More importantly, $500K gross is $300K net. Times 5 is $1.5, or time 10 is $3M. That's pretty good "fuck you" money. On top which some industry street cred allows new faculty to opt out of a lot of the ridiculous BS that happens in academia. Seen this time and again.
I think the easiest and best path for a fresh NLP phd grad can do right now is find the highest paying industry position, stick it out 5-10 years, then return as a profess of practice and tear it up pre-tenure (or just say f u to the tenure track because who needs tenure when you've got a flush brokerage account?)
$100,000 in 1970 is worth almost $800,000 today.
Yes, downvote me all you want. But if you're an NLP expert thinking of working for a company that will make billions off your work, you can and should demand millions at least.
Where is some evidence that NLP is 'solved'? What does it even mean? OpenAI itself acknowledges the fundamental limitations of ChatGPT and the method of training it, but apparently everybody is happily sweeping them under the rug:
"ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows." (from https://openai.com/blog/chatgpt )
Certainly ChatGPT/GPT-4 are impressive accomplishments, and it doesn't mean they won't be useful, but we were pretty sure in the past that we had "solved" AI or that we were just about to crack it, just give it a few years... except there's always a new rabbit hole to fall into waiting for you.
LLMs produce perfectly fluent output and can understand natural language input as well as any human.
However knowledge representation is not solved. We still don't know how to interface a perfect LLM to other systems in the same way a human does things like looking up facts we aren't confident of or using a calculator to do math we cant' do in our head.
These are very significant problems and super important. But they are more adjacent to NLP in the same way tasks like something like Text-to-SQL [1] isn't a pure NLP task.
[1] for example https://github.com/salesforce/WikiSQL
I think LLMs have essentially solved the natural language processing problem but they have not solved reasoning or logical abilities including mathematics.
ChatGPT cannot even reason reliably on what it knows and doesn’t know… it’s the library of Babel, but every book is written in excellent English.
Knowledge representation is a separate problem. NLP gives us some insights into what works here, but the multi-modal aspects of things like GPT4 show there is a lot more to knowledge presentation than just NLP.
I've been asking it about lyrics from songs that I know of, but where I can't find the original artist listed. I was hoping chat gpt had consumed a stack of lyrics and I could just ask it, "What song has this chorus or one similar to X..." It didn't work. Instead it firmly stated the wrong answer. And when I gave it time ranges it just noped out of there.
I think If I could ask it a question and it could go, I've used these 20-100 sources directly to synthesize this information, it'd be very helpful.
https://dkb.blog/p/bing-ai-cant-be-trusted
To answer the question above, these systems cannot provide sources because they don’t work that way. Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training.
They have no way to distinguish or differentiate which piece of the training data was the “actual” or “true” source of what they generated. It’s like the old questions “which drop caused the flood” or “which pebble caused the landslide”.
> Their source for everything is, basically, everything. They are trained on a huge corpus of text data and every output depends on that entire training.
Bing chat is explicitly taking in extra data. It's a distinctly different setup from chatgpt.