If you want my personal opinion, I think the hype around RAG techniques is mostly a fad. I don't see how it becomes a super broad high value technique that your average DS or ML Engineer is getting paid big bucks to know. It's also in danger of becoming totally obsolete with further LLM advances (Claude 3 already really good at needle in the haystack tasks). Not something worth spending a ton of time on unless you're basically already knee deep in that kind of problem space. I think the only reason it's trendy right now is because LLMs are hammers looking for nails and semantic Q&A is one of the few identified nails.

I'm pursuing of a strategy of:

* Doubling down on domain knowledge. Even if coding becomes completely automated, there will always be value in knowing some particular corner of an industry very well. For me, that's insurance. This includes knowing how classical ML techniques can be applied in my industry (LLMs are not replacing xgboost any time soon).

* Keeping tabs on bleeding edge LLM capabilities, just to have a sense of what's possible. However, staying out of highly technical LLM implementation details like CUDA, inference, batching, etc. Perhaps fine tuning will be valuable but probably I would focus on understanding "fine tuning as a service" offerings rather than rolling my own.

* Investing in knowledge of infrastructure/orchestration of LLM services in a way that doesn't tie me to any particular problem space. It's hard to predict right now what kinds of new products LLMs might enable. But I think there will be value in being able to translate business problems into a high level architecture for an LLM-enabled app, even if that just means wiring together existing service offerings.