The downside is that of course the further you stay away from actually playing your instrument, the rustier you will get; this might or might not be an issue, but it is a significant downside, also being away from the technique side means that as the years go by, you might not be able anymore to have an intuitive understanding of "this sounds wrong" and why it does.
I don't want to stretch the metaphor too far, but it does feel that agentic coding is a bit like building skill tech debt, the more you do it, the more you'll have to work later to bring back the skills you lost. The meta could be that models will get so good so that only a small percentage of developers will ever need to sully their eyes again with looking at code (same as while when we started most people had at least a passing familiarity with assembly, nowadays nearly nobody does), jury's out on that.
Either way, what before needed a full orchestra of developers, now could be doable by a good conductor and maybe a couple of soloists: how many orchestras will be needed? and what will the section musicians not good enough to be soloists and that don't have an interest in becoming a conductor will do? and how are new conductors going to develop their technical chops if there are no more orchestral jobs?