That's always been the bar for FOSS, we've just failed as a community at education because we became elitist and decided the muggles didn't actually deserve to understand anything. The end goal of the free software movement should be that everyone has that understanding, that code literacy should be as essential as reading and math. But yes, in short, you don't truly own code you don't understand. Simply having the code available is better than not, but a freedom you can't exercise really isn't a freedom, and all of the freedoms of free software presuppose the ability to comprehend software.
And there's a difference between few being capable of implementing changes, and anyone. In a world where most or all code is written and edited by LLMs, and CS courses are focused on how to prompt LLMs and no longer on fundamentals, the opportunities for humans to learn programming will all but vanish. And this assumes LLMs will even expose code at all. They may simply emit precompiled applications in the future. If humans remain in the loop even when LLMs are used then this becomes less of a problem. But the culture around LLMs converges on vibe-coding - as little direct human understanding or effort as possible. As such it converges on sacrificing freedom for convenience.