I once was obsessed at finding the holy Grail of optimization and scaling, but then came to the conclusion that every system is optimized and scaled differently. Different systems have different bottlenecks.
So true, rarely is anything the “best” or “better”, but instead each thing is a bucket of trade offs we choose from. Maybe frustrating, but also what makes engineering genuinely interesting.
This sounds similar to the no free lunch theorem in AI
Huh? NFL is a theorem on optimisation, not AI
It's originally a theorem about supervised learning, which turned out to be the prototypical application (via CNNs) of deep learning. Deep learning is now mostly synonymous with AI. Though I'm not aware of any NFL theorem for reinforcement learning or unsupervised learning.
Given that the most well-known use case for optimization algorithms is training machine learning models, it seems to me like a perfectly reasonable, if buzzword-oriented, thing to say.