You have to logout to see how others see your profile.

Toggling the 'contributions settings' to 'public only' on my profile still shows me all my private contributions.