When it comes to making a more complicated change, things are never easy because of the limited context window and the general lack of reasoning and inherent knowledge of the codebase I am working with.
Having said this, GPT4 has been really good for the one off questions I have about either syntax or if I forget how to do "the thing I know is possible I am just not sure" or the mundane things like docker commands or some other commands which I need help with.
But... if you guys have seen Gemini-1.5 Pro I was seriously mind blown and I think the first time I felt a LLM is better than me and that has to do with code search. I have had my fair share of navigating large codebases and spending time understanding implementations (clicking go-to-reference, go-to-definition) and keeping a mental model.. the fact that this LLM can take a minute to understand and answer questions about codebase does feel like a game changer.
I think the right way to think about AI tooling for programming is not to ask them to go and build this insane new feature which will bring lots of money for you, but how they can help you get that edge in your daily workflows (small quality of life changes which compound over time, just like how LSP is taken for granted in the editor now a days).
Another point to mention here which I believe is a major miss is that these copilots write code without paying attention to the various tools we as humans would use when writing code (LSP, linters, compilers etc). They are legit writing code like they would on a simple notepad and that is another reason why the quality is often times pretty bad (but copilot has proved that with a faster feedback loop and the right UX its not too big a hassle)
We are still very early in this game and with many people building in this space and these models improving over time I do think we will look back and laugh how things were done pre-AI vs post-smart-AI models.