Reflections from the RAG Funeral: A Melting Pot of Minds and Misconceptions

just as ML engineers of 5 years ago generally knew nothing about SVMs, Gaussian Mixture Models, MLE vs MAP ... ML engineers of today know little about VAEs, GANs, LSTMs, Reinforcement Learning.

Cold topics of yesteryear quickly fall off the map.

I would go further and claim that a 2024 ML Engineer may well be unaware of 2019 BERTology and GPT internals as well as 2020 fast Attention kernels and be today focused solely on: a) Huggingface pipeline SFT, Diffusion, and Langchain RAG chunk GenAI orchestration. b) the ever-prevalent mind-numbing drudgery of data munging that occupies 50% of a data scientist's time.

I mean, all that effort put into learning PyTorch / Tensorflow implementations of Transformer architectures - is it relevant without 10,000 TPU year resources, when big tech GPU hoarders expose simple GenAI REST APIs for mass consumption ? An analogy: learning how CPUs are optimally engineered, when you dont have access to a fab, is completely irrelevant to app dev. a complete bifurcation of skills between the laymen and the high priests. OOTOH, if AGI ever takes off, it may be advantageous to have some idea of how it works.