You don't have to add them back. Just like twitter, following/circling is not a symmetrical operation.

Those people will only see your updates if you send something that is public and is not to a circle.