What do you mean with "building this stuff"? As in building LLMs, or building applications on-top of them.

I find this obsession with building strange. It's a very SaaS Silicon Valley mindset. There are whole swathes of very talented engineers who spend most of their time debugging, characterizing systems, doing performance analysis and resolving bottlenecks. Some of it might require writing significant code but mostly it's writing small test cases. The key skill is to treat a computing system as the object of study and to be a good empirical scientist (which requires understanding theory pretty well). These are people with deep expertise in networking, GPUs, CPUs, memory etc. One only has to look at national labs that do large-scale HPC (high-performance computing) to see examples.

One can argue that a lot of "building with AI" is commoditized by fine-tuning and RAG libraries or even reduced to prompt engineering. A lot of it is also tricks that might work on one dataset but not others. Putting together libraries fueled by pizza and coke gives an illusion of skill and speed.

Are there grifters who are jumping onto the AI bandwagon? Of course! In spades. Are there also engineers who want to build up their skills and are failing to do so or in the process of doing so? Of course, this happens too! But there are also people who are trying to understand, debug and improve models who are not necessarily "building". After all, the scaling laws paper (the original one) was a result of pure analysis of empirical data.

No, it is being able to evaluate models. 5 builders without eval produce zero value. 1 eval person can pick and choose the best model out of a bunch that are open source and commercial and maybe one of them is good enough. Put an eval person together with N builders and you have a chance of making a good enough model.

If you want to know why Hacker News is full of people disappointed or skeptical with AI ask yourself why they put 99.9% of their effort into “zero-shot” when it is clear as day that if you get a few thousand examples and train in that you wipe the floor with “zero-shot”