For an individual copyright holder, or software which is static and no longer being developed, you are correct.
However if either of those preconditions fails, then the reverse becomes true "in practice."
When the copyright is held by multiple parties, re-licensing becomes impractical (or impossible) because every copyright holder needs to agree to the new terms. The more of them there are, the more likely it is that someone will veto.
However, because the AGPL and Apache licenses (and other liberal licenses, MIT, BSD, etc.) are compatible, I can take liberally licensed software, add loads of fancy features using an AGPLv3 license and the combined end result will become AGPLv3 licensed in practice. Just like the combined license of OS X or Chrome is proprietary, even if it contains open source components. The most restrictive license wins.
So assuming the software is evolving over time and you have a wide base of contributors, it is possible to go from less restrictive to more restrictive, but not the other way around.