As I understood, the circles feature is especially suited to solve such "cant unfriend" problems, by letting you create an entirely similar circle but without the person you want to politely get rid of, and simply never tell him, so he'll never find out and you can graciously avoid the blame of "unfriending" someone.

I don't think he knows when you remove him from a circle, I don't think he knows the name of the circle too.