Neither impossible nor tough, but also not a clean way of managing state. You need to keep track of all past things you once did but no longer need.
Not trying to discourage innovation and enthusiasm. I just want my router setup.
It feels very inconsistent to me. You can always start from a minimalist system image and then add a few packages you need using Ansible. Is that really that different?