> I'd really love someone to describe in detail the type of programming they are doing that can't be codified such that another human can learn it, follow the rules and achieve the needed results. I just don't see any magic in our craft, it is all just following rules.
This is not magic, this is true, but it's deep and very niche knowledge. Let me give an example from my Ph.D., where I did some low level, high performance programming in Boundary Element Method space.
The knowledge is not novel, but the formulae is. We developed the math, not optimized something already out there. Moreover, we had to optimize to the hardware architecture we had. This means tons of runs, profiles, optimizations, and even more runs.
There are some bottlenecks here. You can't make profiling faster since you're already going flat out. Memory bandwidth, processor's internal pipelines, load and store units are completely saturated. Perf returns numbers close to theoretical maximums, the systems are running at TDP limits, you're done. AI can't make it faster.
Developing the math, chopping the formula and sprinkling at different levels of the loop to minimize step count to use what you have at hand becomes important. You also check modeling accuracy here, testing around 32 significant digits precision, again takes time.
If you're changing processor architectures, load/store widths change; pipelining behavior change, memory bandwidth per core, NUMA structure change. You have to fine tune here and there to get the same efficiency from a different core.
So, method is not the bottleneck, but the novelty of the problem and method and runtime is. When testing engineering stuff precision and accuracy both matters, and seeing tradeoffs take time.
When there's nothing to draw from, maybe AI can point out blaring issues, but without running the code and seeing it for yourself, you can't reach to the point where you need to go.
For the "codification of it" part, humans have something called intuition which is a kind of tacit knowledge which shows us the way based on a wide network of knowledge. It's not easy to surface, define, codify and transfer. That knowledge esp. helps when systems act contrary to guesswork and rules break in myriad of ways. "Having a feeling of the machine" is only possible with experience, and can't be codified and transferred easily. This is why we have master/apprentice model and why it's so important in transferring knowledge.