The primary push that OpenAI and most other AI-hype companies now are talking about, to improve performance, is to do the same thing but bigger. No doubt there will be architectural changes (certainly many are in fact needed), but that is hard to predict, and "make it bigger" is easier to plan for.

So, going from GPU's to some more focused chip architecture could reduce the power usage some, but more likely is that the total power usage will go up not down, as these companies try to address any problem with bigger scale. Not saying it will work, but it is the most likely thing to be tried, and it will take several years before they give up on that route.