If the person that you followed blocks and then unblock you, you are removed from their followers list.

Could that be the case in your instance?