You're not supposed to stop using other peoples castle, You're just supposed to have a castle of your own to bring people to.