I think this is worth thinking about but I would be less eager to say it is obvious.

Our own notion of agency starts to look rather illusory if you look hard enough, and big questions remain regarding free will or the lack thereof.

It is possible to look at agency as just another experiential aspect of consciousness - something we feel, like many other feelings - but that feeling could be just as involuntary as any other feeling we experience.

I think the lack of clear evidence for and understanding of free makes agency a poor measure for AI progress.